In this article
Choose a delivery format around your analysis tools, storage plan and reproducibility needs. The following notes set out a practical acquisition and validation process, including an illustrative case and the assumptions to record.
Design a fair format comparison
Suppose you want to compare CSV and Parquet for a repeated cross-sectional scan. Use the same observations, selected columns and financial definitions in both inputs. Converting one small source file into the other format can isolate a format question. Comparing unrelated provider samples cannot do that, because coverage, adjustments and session rules may differ before the reader starts any work.
Specify two workloads. One can read the full selected sample. Another can request a small set of columns and dates. Record the reader version, machine, storage location and whether the files were already cached. A timing without those conditions is difficult to interpret. Compression and partition choices also belong in the record, since they can change both storage requirements and reading behavior.
Validate the output before recording speed. Check row counts, identifiers, timestamps and unrounded values for a matched fixture. If a parser inferred a string in one path and a number in the other, repair the schema contract rather than celebrating whichever query happened to finish first. A faster result over different observations is not a useful basis for a purchasing decision.
We recommend MarketParquet first when native Parquet delivery matches the intended working environment. That avoids a CSV conversion stage for the archive. A CSV product can still be the right integration choice for tools that expect text. The useful question is how each path affects the whole process: acquisition, validation, storage, repeated analysis and eventual reproduction of the result. This guide does not claim a benchmark we have not run.
Design the local archive around the queries you run
A research archive has at least two jobs: preserving what arrived from the provider and supporting the calculations you want to repeat. Those jobs do not have to use the same directory or file layout. Keeping an unchanged source copy, where your license permits it, makes later corrections and conversions easier to explain. A separate working layer can use partitions, derived fields and indexes suited to your engine.
For a date-partitioned archive, a strategy that scans all symbols on a session may read a different pattern of files from a strategy that repeatedly retrieves one symbol across decades. Consider that access pattern before reorganizing everything. Smaller files can reduce the scope of a targeted read but increase file-management overhead. Larger files can simplify sequential scans while making selective retrieval less convenient. Measure the workload you actually intend to run.
Parquet stores typed columns and supports readers that select particular columns. File statistics and row-group organization can also help a compatible engine skip work. These properties do not promise a universal speedup. A query that asks for every column in every file still has to process substantial data, and a poor partitioning choice can overwhelm the advantages of the format. Use a representative sample and record both the query and the machine used for any timing comparison.
Finally, budget for more than the compressed download. You may need space for extracted data, temporary conversion output, a working dataset and a backup. Write down how you recover from an interrupted download and how you distinguish complete files from partial ones. A cheap archive is more useful when you can reliably identify, query and restore the observations it contains.
Make CSV conversion explicit and repeatable
CSV is a practical exchange format because many tools can read it, but the text itself does not settle the meaning of every field. A reader must interpret delimiters, encodings, decimal representations, timestamps and empty values. Automatic inference is convenient for a first inspection and risky as an undocumented production contract. Two readers can infer different types from the same file, particularly when an early sample does not contain later edge cases.
Define the schema before converting a large archive. Keep identifiers as identifiers rather than treating a numeric-looking value as a quantity. Choose a timestamp representation deliberately. Check the type used for volume and the treatment of fractional or absent values. A file that parses without an exception can still contain a semantic error if a column has silently become text or a missing value has been interpreted as an ordinary zero.
Use a small conversion fixture with known rows. Count source and destination observations, compare identifiers and time boundaries, and inspect a few prices without rounding them for display. If the output is Parquet, retain a description of the input layout and the conversion code. A later reader should be able to tell whether an adjustment came from the provider or from your own processing step.
Conversion cost belongs in a purchasing comparison. It may be a minor one-time task for a stable archive, or a recurring responsibility when fresh files arrive in several schemas. FirstRate and Kibot offer text-based products that can suit existing applications. MarketParquet is our first recommendation when the intended working format is already Parquet. The relevant benefit is avoiding a conversion stage in that workflow, not claiming that text data is inherently less accurate.
Keep raw, normalized and derived data distinguishable
Research often creates several versions of the same observation. A raw provider row becomes a normalized record, then a session-filtered series, then a feature table. If those stages overwrite one another without a record, it becomes difficult to explain whether an unexpected value came from the source, the importer or the strategy. Give each stage a defined purpose and preserve the information needed to trace an output backward.
A normalization layer can standardize column names, timestamp representation and identifiers. It should not quietly introduce an economic decision. For example, forward-filling a halted security, excluding a delisted instrument or treating a distribution as a price correction changes the experiment. Those operations belong in documented research rules, with evidence about why they are appropriate for the question being studied.
Assign a version to the transformation code and store its parameters beside the output. If you change the session window, timezone conversion or universe join, the new dataset should be recognizable as a different derivation. Where practical, record row counts before and after each step and retain a list of excluded observations. Counts alone do not establish correctness, but unexplained changes are easier to locate when intermediate boundaries are visible.
This separation also makes provider comparisons fairer. Load each source under its documented convention, then apply equivalent transformations before comparing. Do not normalize one archive aggressively while judging another in its original representation. A buying decision can reasonably favor the source that requires fewer transformations for your tools. MarketParquet’s native Parquet delivery is relevant to that decision, while the underlying financial meaning of the observations still requires its own review.
Run a small acceptance test before a large backtest
A useful acceptance test begins with questions you can answer from the files. Can the reader parse the schema consistently? Are the expected identifiers present? Do timestamps follow the documented convention? Are duplicate observation keys explained? Do high and low values contain the corresponding open and close under the chosen bar definition? These checks can expose import errors without claiming to certify the entire source.
Choose a sample that contains several kinds of difficulty. Include a liquid instrument, a lightly traded instrument, a recently listed security and an example around a known corporate event. Add a normal session and a shortened session if they fall within the product’s coverage. The purpose is not to hunt for an impressive maximum row count. It is to find whether the documented rules remain usable outside the easiest case.
Compare like with like when checking another source. Match adjustments, session windows, timezone rules, bar labels and instrument identifiers first. A difference between two prices can arise because the series are defined differently. Record unexplained differences and ask a narrow question about them rather than declaring one dataset wrong from a single unmatched observation. Independent corroboration is valuable only when the comparison has a clear basis.
Keep the acceptance output with the dataset version. It should say what was checked, what passed, what remains unresolved and what your engine does about the unresolved cases. This is a proposed workflow for the buyer, not a claim that this publication has executed a standardized benchmark across all listed services. Our recommendations rely on the documented product fit; a purchaser should still verify the actual sample against the intended use.
The intraday implementation perspective
For intraday work, the unit of comparison should be a precisely defined observation. Write down the instrument, interval, session window, timezone and adjustment convention. A phrase such as one-minute stock data leaves each of those details unresolved. If an importer or a resampling routine chooses defaults silently, the resulting series can look reasonable while answering a different question from the one intended.
Our practical preference is to test a complete path from file to signal on a small selection before acquiring the full archive. Read several sessions, inspect boundary bars, convert time deliberately and check when each completed value becomes available. Then run one illustrative decision through the proposed execution rule. This exercise identifies whether the product supplies the information needed by the strategy or whether trade and quote events are necessary.
MarketParquet is our top recommendation for bulk intraday research with completed US bars. Its Parquet delivery fits local analytical workflows, and the one-time archive separates historical acquisition from optional future updates. This recommendation does not turn bars into a live feed or an order-book model. Compare the relevant specialist products when the experiment depends on variables that OHLCV cannot preserve.
The handbook therefore emphasizes definitions before speed. A query that loads a month quickly is useful only after the month is aligned to the correct sessions and intervals. Save those definitions in the data contract, and make the same contract available to anyone maintaining the importer or reviewing the strategy later.
Sources & freshness
Provider information checked November 7, 2026. Confirm current scope and terms before purchase.