this is gonna be long. grab water. i wrote this over the week i had packets flying over loopback from real NASDAQ data, so everything is still fresh in my head.
I had zero finance knowledge a week ago.
Not like "oh I know the basics but not the advanced stuff." I mean genuinely zero. I didn't know what a bid was. I didn't know what an ask was. I thought HFT was just some fancy term for people who trade fast.
Then a friend explained order books to me in a voice message and something clicked. The idea that somewhere, right now, there's a system processing 74 million market events in a single trading day, in nanoseconds, with no mutex, no heap allocation, no kernel involvement if it can help it... that was enough. I wanted to build that.
So I built PHOTON. A nanosecond market data feed handler in C++ that ingests real NASDAQ ITCH 5.0 binary protocol over UDP, parses it with zero copies, and publishes to a lock-free SPSC ring buffer. The PRD target was under 300 nanoseconds end-to-end at p50. Parsing alone came in at 27 nanoseconds, way under target. End-to-end landed higher, and the reason why turned out to be one of the more interesting things I learned building this.
This post is everything I learned. The finance, the systems, the C++ I had to actually understand before I could write it.
First, what even is a stock exchangeCopied!
A company needs money. Instead of taking a loan, it sells tiny pieces of ownership to the public. Each piece is a share. You buy shares, you own a fraction of the company, and if the company grows, your shares are worth more.
NASDAQ and NYSE are the two big exchanges in the US. They're the matchmakers. You want to buy Apple, I want to sell mine, we both go through NASDAQ.
When you want to buy or sell, you place an order. Two kinds matter here:
Market order -- buy right now at whatever price. Guaranteed to happen, price unknown.
Limit order -- buy only if price is at or below X. You control the price. Not guaranteed to happen if nobody's selling at your price.
HFT firms use limit orders almost exclusively. Precision over speed of execution.
The order book, which is what my friend builtCopied!
The exchange keeps an "order book" for every stock. It's a live list of every limit order that hasn't been matched yet. Two sides:
Bids -- people wanting to buy, sorted highest price first.
Asks -- people wanting to sell, sorted lowest price first.
ASKS (sellers)
$189.52 -> 500 shares
$189.51 -> 1,200 shares
$189.50 -> 800 shares <- best ask
---------------------------------
$189.49 -> 2,000 shares <- best bid
$189.48 -> 750 shares
BIDS (buyers)The gap between best bid and best ask is the spread. When a buy order comes in at $189.50, it matches the best ask, a trade happens, both orders get removed. If it comes in at $189.45 and nobody's selling that cheap, it just sits in the bid side and waits.
My friend built the matching engine -- the thing that maintains this book, matches orders, and handles the data structures (he used a fixed-size array indexed by price for O(1) lookup instead of a sorted tree, which is genuinely clever). I built the layer before that. The thing that receives raw bytes from NASDAQ and hands clean, parsed events to the matching engine.
Where PHOTON sits in the actual pipelineCopied!
NASDAQ exchange
| ITCH 5.0 UDP stream
v
PHOTON
| ParsedEvent structs in a ring buffer
v
Order Book / Matching Engine
| current market state
v
Strategy (decides when to trade)
| order decision
v
Order Router
| back to NASDAQ
v
NASDAQ exchangePHOTON is the feed handler. It doesn't make decisions. It doesn't know what to trade. Its only job is: receive bytes, parse them, deliver structured events to the next layer as fast as physically possible.
The reason this matters: if my PHOTON delivers an event in 200ns and yours delivers it in 500ns, your matching engine sees the price change 300ns later than mine. In HFT, 300ns is enough time to place and confirm an order. I already won before you even started.
Why these firms colocate (the geography problem)Copied!
Speed of light through fiber optic cable from California to New York takes about 40ms. If you're in California and a firm is in New Jersey next to the NASDAQ datacenter, their order arrives 40ms before yours. In HFT, you lose 100% of the time.
So firms like Tower Research and DE Shaw rent server racks inside NASDAQ's own building in Carteret, New Jersey. Their servers are literally 30 meters from NASDAQ's matching engine. The latency is determined by the length of the ethernet cable, not geography. NASDAQ guarantees all colocation cables are the exact same length to the millimeter so nobody gets a hardware advantage.
There's also the microwave tower story. Light travels 30% slower inside fiber optic glass than through open air, and cables have to bend around geography. HFT firms built private microwave tower networks in straight lines across mountains from Chicago to New York. Radio waves travel at the speed of light through air. They beat fiber by a few milliseconds on a clear day. If it rains hard or a bird flies in front of the dish, signal drops. But when it works, they print money.
After all that physics, the last battleground is software. That's where PHOTON lives.
What NASDAQ ITCH 5.0 actually isCopied!
Every event on NASDAQ -- someone places an order, cancels one, a trade executes -- gets broadcast as a binary UDP packet to every connected participant simultaneously. The protocol is called ITCH 5.0.
Binary means not text. No JSON, no XML, no CSV. Raw bytes in a specific layout, big-endian, packed tight. The four message types PHOTON's parser was built around:
A-- Add Order. Someone placed a new limit order. Contains stock ticker, side (buy/sell), shares, price, a unique order reference number.E-- Order Executed. A previously placed order got matched. Contains the order ref number linking back to the Add Order.X-- Order Cancel. Someone cancelled part or all of a waiting order.P-- Trade. A trade happened.
Every HFT firm receives this exact stream simultaneously. The one that processes it fastest and acts first wins.
The system: three processes, three coresCopied!
[CORE 0] exchange_sim
reads real NASDAQ ITCH binary file
replays packets as UDP multicast to 239.1.1.1:30001
configurable speed: 1x, 10x, 100x real-time
[CORE 1] photon_receiver
SO_BUSY_POLL socket -- spins, never sleeps
recvmmsg -- receives 256 packets per syscall
zero-copy parse: reinterpret_cast<ITCHMessage*>(buf)
rdtscp timestamp at receive and at parse
pushes ParsedEvent into SPSC ring buffer
[CORE 2] photon_consumer
pops ParsedEvent from ring buffer
rdtscp timestamp at consume
computes parse_latency = ts_parsed - ts_received
computes total_latency = ts_consumed - ts_received
accumulates into latency histogram
prints p50/p95/p99 every secondNo external libraries. No heap allocation during operation. No mutex anywhere. Total external dependencies: zero.
The clock: why rdtscp and not clock_gettimeCopied!
The first thing I built was the timing system, because every latency measurement in the project depends on it.
Your CPU has a register called the TSC -- Time Stamp Counter. It's a 64-bit number that increments once per CPU cycle, forever, since boot. On my i7-14700HX running at ~5.5GHz, one tick is about 0.18 nanoseconds.
clock_gettime(CLOCK_MONOTONIC) is how you normally get time in Linux. It costs ~25 nanoseconds per call because it has to switch to kernel mode, read a hardware clock, convert units, and switch back. That's 25ns of overhead just to measure time. My parse ended up taking 27ns -- if I'd used clock_gettime to measure it, the timer itself would have cost nearly as much as the thing it was measuring.
rdtscp is a single CPU instruction. Reads the TSC register into a 64-bit integer. Costs ~4 nanoseconds. The p stands for serializing -- it waits for all prior instructions to complete before reading, so out-of-order execution can't corrupt the measurement.
The calibration function measures how many TSC ticks equal one nanosecond on this specific CPU right now:
double calibrate_tsc_ghz() {
uint64_t t1 = rdtsc_now();
struct timespec duration;
duration.tv_sec = 0;
duration.tv_nsec = 100'000'000; // 100ms
clock_nanosleep(CLOCK_MONOTONIC, 0, &duration, nullptr);
uint64_t t2 = rdtsc_now();
return (double)(t2 - t1) / 100'000'000.0;
}Use the slow OS clock (accurate over long durations) to calibrate the fast TSC clock (accurate per-tick). Run it once at startup. My machine measured 2.3059 GHz -- meaning one tick is 0.4337 nanoseconds. Every latency number in my histogram uses this.
Zero-copy parsing with reinterpret_castCopied!
Normal parsing: receive bytes, copy into a string, parse the string, produce a struct. Every copy costs time and memory bandwidth.
Zero-copy: receive bytes, cast those bytes directly to a struct. One pointer operation. No copy, no loop, no allocation.
This works because #pragma pack(1) tells the compiler to put no padding between struct fields. The struct layout matches NASDAQ's wire layout byte for byte. So:
// The packed struct mirrors the wire exactly
#pragma pack(push, 1)
struct AddOrderMsg {
uint8_t msg_type;
uint16_t stock_locate;
uint16_t tracking_number;
uint8_t timestamp[6];
uint64_t order_ref;
uint8_t side;
uint32_t shares;
uint8_t stock[8];
uint32_t price; // dollars x 10000. $189.47 = 1894700
};
#pragma pack(pop)
// Zero-copy parse: no allocation, no copy
inline const AddOrderMsg* parse_add_order(const uint8_t* buf) {
return reinterpret_cast<const AddOrderMsg*>(buf);
}One thing: NASDAQ sends multi-byte integers big-endian (most significant byte first). x86 CPUs are little-endian (least significant byte first). So $1000 arrives as bytes 00 00 03 E8 but your CPU reads that as E8 03 00 00 = 232456. Fix with __builtin_bswap32() -- a single bswap CPU instruction, zero overhead.
Prices are always integers. AAPL at $189.47 arrives as 1894700. You only divide by 10000 when printing for humans. Floating point arithmetic on financial data introduces rounding errors. Integers are exact.
The ring buffer: lock-free, cache-line awareCopied!
The receiver thread (Core 1) and consumer thread (Core 2) need to share data without a mutex. A mutex would block -- blocking adds latency. Instead: a lock-free SPSC ring buffer.
SPSC = Single Producer Single Consumer. One thread writes, one reads. This constraint eliminates all need for locks.
template<size_t N>
class SPSCRingBuffer {
static_assert((N & (N - 1)) == 0, "N must be power of 2");
alignas(64) std::atomic<uint64_t> head_{0}; // producer owns this
alignas(64) std::atomic<uint64_t> tail_{0}; // consumer owns this
alignas(64) ParsedEvent slots_[N];
public:
bool push(const ParsedEvent& event) {
uint64_t h = head_.load(std::memory_order_relaxed);
uint64_t t = tail_.load(std::memory_order_acquire);
if (h - t == N) return false; // full, drop event
slots_[h & (N - 1)] = event;
head_.store(h + 1, std::memory_order_release);
return true;
}
bool pop(ParsedEvent& out) {
uint64_t t = tail_.load(std::memory_order_relaxed);
uint64_t h = head_.load(std::memory_order_acquire);
if (h == t) return false; // empty
out = slots_[t & (N - 1)];
tail_.store(t + 1, std::memory_order_release);
return true;
}
};A few things worth understanding here:
Power of 2 size: h % N is a division, ~20-40 CPU cycles. h & (N-1) is bitwise AND, 1 cycle, same result when N is power of 2. The static_assert enforces this at compile time.
False sharing: head_ and tail_ are written by different threads. If they shared a 64-byte cache line, every write by the producer would invalidate the consumer's cache and force a re-fetch from RAM. alignas(64) forces each onto its own cache line. They never touch each other's memory.
Memory orders: release on a store means "all my prior writes are visible before this index update." acquire on a load means "I see everything the other thread wrote before updating this index." Together they form a happens-before guarantee without a mutex. Not cargo cult. Actually required for correctness on modern CPUs that reorder memory operations.
ParsedEvent is exactly 64 bytes -- one cache line. Enforced with static_assert(sizeof(ParsedEvent) == 64). If you accidentally add a field and push it to 65 bytes, every ring slot straddles two cache lines and latency doubles. The assert catches it at compile time before you ever run anything.
The receiver: where everything meetsCopied!
This is the part that actually touches the network, and it's where most of the latency decisions live.
Batch receiving with recvmmsg. A normal recvfrom call does one syscall per packet. Syscalls cost ~100ns each from the user/kernel mode switch alone, before you've even touched the data. At exchange speeds that adds up fast. recvmmsg receives up to 256 packets in a single syscall, amortizing that 100ns over the whole batch instead of paying it per packet.
struct mmsghdr msgs[256];
// ... fill in iovecs pointing at pre-allocated buffers ...
int n = recvmmsg(sock_, msgs, 256, MSG_DONTWAIT, nullptr);SO_BUSY_POLL instead of blocking. A normal blocking socket sleeps until the kernel wakes it with an interrupt, then the scheduler has to context-switch your thread back in. Both of those cost real time and neither is predictable. SO_BUSY_POLL tells the NIC driver to poll for packets directly instead of waiting for an interrupt, and combined with spinning in userspace on recvmmsg, the receiver thread never actually sleeps. It burns a full core doing this, which is exactly the trade you want when Core 1 is pinned and dedicated.
Joining the multicast group. NASDAQ broadcasts ITCH over UDP multicast, so before any of this works the socket has to explicitly join the group:
struct ip_mreq mreq;
mreq.imr_multiaddr.s_addr = inet_addr("239.1.1.1");
mreq.imr_interface.s_addr = INADDR_ANY;
setsockopt(sock_, IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq, sizeof(mreq));Without this the kernel just drops the multicast traffic at the network layer before it ever reaches the socket buffer.
The parse switch. Once a packet is in hand, a switch on the first byte (the message type) dispatches into the right zero-copy cast and builds a ParsedEvent:
switch (buf[0]) {
case 'A': {
const auto* msg = parse_add_order(buf);
event.type = EventType::AddOrder;
event.order_ref = __builtin_bswap64(msg->order_ref);
event.price = __builtin_bswap32(msg->price);
event.shares = __builtin_bswap32(msg->shares);
break;
}
// 'E', 'X', 'P' follow the same shape
}Message types I didn't plan for. The PRD scoped four message types: A, E, X, P. The real NASDAQ feed has a lot more than that, and once I pointed PHOTON at actual exchange data I started seeing D (Order Delete), U (Order Replace), and L (Market Participant Position) show up in the stream. None of those were in scope. The switch has a default case that just increments an "unhandled message type" counter and moves on instead of treating it as an error -- a feed handler that crashes or stalls on a message type it doesn't recognize is worse than useless in production, since the exchange isn't going to stop sending them for your benefit. Handling the unknown gracefully turned out to matter more than handling every known type.
The consumer: closing the loopCopied!
Core 2 runs the consumer, and its only job is to drain the ring buffer as fast as the receiver fills it.
ParsedEvent event;
while (running) {
if (ring.pop(event)) {
uint64_t ts_consumed = rdtsc_now();
uint64_t parse_latency = event.ts_parsed - event.ts_received;
uint64_t total_latency = ts_consumed - event.ts_received;
histogram.record(parse_latency, total_latency);
}
// no sleep, no yield -- spin and check again
}Same spin-don't-block philosophy as the receiver. ts_received gets stamped the moment recvmmsg hands back the packet, ts_parsed the moment the struct cast finishes, and a fresh rdtscp call at consume time gives ts_consumed. Subtracting those gives the two numbers that actually matter: how long the parse itself took, and how long the full pipeline took from wire to consumer thread.
The one tricky bit is the histogram report timer. Printing a report every second sounds like it wants a sleep(1) somewhere, but sleep inside the hot loop would mean the consumer stops draining the ring buffer for that second, which is exactly the kind of blocking the whole design exists to avoid. Instead the loop checks elapsed rdtscp ticks against the calibrated GHz value on every iteration, and only prints when a full second's worth of ticks has actually passed. The loop never blocks; it just occasionally notices that time has passed.
The exchange simulator: replaying real NASDAQ dataCopied!
Before I could test the receiver, I needed something sending real ITCH packets. exchange_sim reads the actual NASDAQ historical binary file and replays each message as a UDP packet to multicast address 239.1.1.1:30001.
The ITCH file format is simple: each message is prefixed with a 2-byte big-endian length, followed by that many bytes of message. Read 2 bytes, swap endianness, read N bytes, send over UDP. Repeat for 74 million messages.
while (true) {
uint16_t length;
if (fread(&length, sizeof(length), 1, file) != 1) break;
length = __builtin_bswap16(length);
uint8_t buffer[65535];
if (fread(buffer, 1, length, file) != length) break;
sendto(sock_, buffer, length, 0,
(struct sockaddr*)&dest, sizeof(dest));
}The moment I got this working and ran tcpdump on loopback:
21:55:17.054698 IP fedora.48836 > 239.1.1.1.pago-services1: UDP, length 12
21:55:17.054714 IP fedora.48836 > 239.1.1.1.pago-services1: UDP, length 39
21:55:17.054718 IP fedora.48836 > 239.1.1.1.pago-services1: UDP, length 39
...
10 packets captured
240054 packets received by filter
227573 packets dropped by kernel227,573 packets dropped by kernel. tcpdump couldn't keep up logging them. The exchange sim was sending faster than the logging thread could print. That number right there tells you why you need a busy-poll receiver, not a normal blocking socket.
The latency histogram: what winning looks likeCopied!
Every second, PHOTON prints something like this:
=== PHOTON LATENCY REPORT (1s window) ===
Events processed : 57,142
Parse latency : p50= 27ns p95= 39ns p99= 46ns p99.9= 56ns
Total latency : p50= 468ns p95= 492ns p99= 753ns p99.9= 1,244ns
Ring drops : 0Terminal Output:
The histogram uses 10,001 buckets, one per nanosecond from 0 to 10,000ns. Anything above that goes into an overflow bucket -- those are tail latency events caused by kernel interrupts, thermal throttling, or context switches. Finding p99 doesn't require sorting: walk the bucket array from index 0 accumulating counts, stop when you've hit 99% of events. O(10001) scan, ~10 microseconds to compute.
Two latency numbers matter:
parse_latency = ts_parsed - ts_received: how long the parse itself took, from socket receive to ParsedEvent ready.
total_latency = ts_consumed - ts_received: end-to-end, from network to consumer thread reading the event.
Both measured with rdtscp ticks converted to nanoseconds using the calibrated GHz value.
Now the honest part. Parse latency at 27ns absolutely demolished the PRD target of 187ns -- the zero-copy cast plus a single switch statement is just fast, there isn't much left to optimize there. Total latency at 468ns p50 is a different story. The PRD target was 300ns, and I came in well above it.
The reason is the batching itself. recvmmsg pulls up to 256 packets per syscall, which is exactly what makes the receiver fast. But the consumer pops one event at a time off the ring buffer. So picture a batch of 256 packets landing in one syscall: event 1 in that batch gets parsed and pushed almost immediately, but event 256 has to wait for the other 255 to get pushed first, and then wait again for the consumer to work its way down the ring buffer to reach it. That queueing delay inside the batch is most of the gap between 27ns parse and 468ns total -- it's not the parse that's slow, it's the wait in line before the consumer gets to you.
Zero ring drops is the number that actually matters most here. It means that even with that queueing delay, the consumer never fell behind the producer over the full run -- the ring buffer never filled up and PHOTON never silently lost an event. The latency tax from batching is real, but correctness held.
What I actually learnedCopied!
I came into this knowing C++ basics and Linux basics. Here's what I know now that I didn't before:
On C++: reinterpret_cast and when it's valid, #pragma pack, std::atomic with explicit memory orders, alignas, template classes, static_assert, the difference between inline functions and real function calls, and why new is banned in the hot path.
On Linux: UDP sockets, setsockopt with SO_BUSY_POLL and SO_RCVBUF, recvmmsg for batch receives, sched_setaffinity for core pinning, mmap, how /proc/cpuinfo flags like constant_tsc matter, and joining a multicast group with IP_ADD_MEMBERSHIP.
On computer architecture: cache lines, false sharing, the TSC register, CPU frequency scaling, L1/L2/L3 cache hierarchy, memory ordering, why out-of-order execution exists and what serializing instructions do.
On debugging things that have nothing to do with logic errors: real linker errors. "Multiple definition" of a function I'd written in a header because I forgot the inline keyword, so every translation unit that included it generated its own copy and the linker had no idea which one to keep. Undefined references because a function was declared and used but never actually compiled into the binary I was linking -- turned out it lived in a .cpp file that wasn't in the build target. Neither of those errors tells you what's actually wrong, they just tell you the linker is upset, and you have to reason backward from there.
And the tcpdump moment was its own lesson. I ran it expecting a clean log of packets and instead watched it report hundreds of thousands of packets dropped by the kernel because the logging couldn't keep up with traffic running anywhere close to real exchange speed. That's not a bug, that's the entire reason a feed handler has to busy-poll and avoid syscalls in the hot path -- even a tool as simple as packet logging falls over at this rate if it isn't built for it.
On finance: what a stock is, what an order book is, how bids and asks match, what a market maker does, why the spread exists, what NASDAQ ITCH is and why HFT firms care so much about parsing it faster than the competition.
The most important thing I learned isn't on that list. It's that understanding every line before you write the next one is the only way to actually own what you build. I wrote every meaningful line of this project myself. I got stuck, I read man pages, I cried a little, I figured it out. The receiver, the exchange sim, the calibration -- mine.
That's the only version of this that's worth anything.
PHOTON is functionally done -- exchange sim, receiver, consumer, ring buffer, and the latency histogram all running end to end on real NASDAQ data. If you're building something similar or have questions, reach out at @xevrion.