OsmSharp

The PBFCompleteWayReader avoids the massive memory overhead of a global node coordinate dictionary by processing the PBF in discrete batches. This is achieved through the following lifecycle:

1. Way Collection: The reader iterates through ways using MoveNext(OsmGeoType.Way, lastId + 1). It buffers way metadata and tags and accumulates the required node IDs into a HashSet<long>.
2. Budget Trigger: Once the number of unique node IDs in the current batch reaches the nodeBudget, the collection phase stops.
3. Node Fetching: The reader calls source.Reset() and uses MoveNext(OsmGeoType.Node, sortedId) to perform a monotonic walk of the required node IDs. Because PBFOsmStreamSource maintains a block index, it can skip entire blobs that do not contain the requested IDs, making this second pass efficient.
4. Materialization: Each fetched Node is stored in a dictionary. The reader then constructs CompleteWay objects by combining the buffered way data with the fetched node data.
5. Cleanup: After the batch is yielded, the way buffer, node ID set, and node dictionary are discarded. The process repeats for the next range of ways.

Note on Performance: The first batch may have a higher 'collect' time because it warms up the PBFOsmStreamSource block index by walking the node section. Subsequent batches benefit from a warm index and faster skips.

The PBFOsmStreamSource supports a persistent block-index sidecar that allows for fast, targeted jumps to specific OSM primitive types (Nodes, Ways, or Relations) without decoding unnecessary blobs.

When persistBlockIndex is set to true, the source builds an index as it walks the PBF. This index contains:

• Tier-0: File offsets (start and end) for each blob on disk.
• Tier-1: Presence of primitive types (HasNodes, HasWays, or HasRelations).
• Tier-2: Min/max IDs for the types present.

On Dispose(), the index is written to a file named {pbfPath}.blockindex. Subsequent runs with the same PBF will automatically load this sidecar, allowing the source to seek past irrelevant blobs without decompression.

// Persist the accreted block index on Dispose() and periodically during the walk.
using var src = new PBFOsmStreamSource(pbf, persistBlockIndex: true);
foreach (var geo in src) { /* ... */ }

When streaming from OSM-PBF or OSM-XML files, ways and relations only contain references to nodes or members by their IDs. OsmSharp provides the ToComplete() extension method to transform a stream of partial objects into a stream of complete objects, where the referenced nodes or members are actually included in the object.

Important Memory Warning: Using ToComplete() requires keeping the referenced nodes/members in memory. To avoid OutOfMemoryException or excessive memory usage, you must apply all necessary filters (e.g., filtering by type or tags) to your stream before calling .ToComplete(). This ensures only the required nodes are held in memory.

using (var fileStream = File.OpenRead("path/to/file.osm.pbf'))
{
  // create source stream.
  var source = new PBFOsmStreamSource(fileStream);
  
  // filter all powerlines and keep all nodes.
  // IMPORTANT: Filter BEFORE calling ToComplete() to manage memory.
  var filtered = from osmGeo in source
   where osmGeo.Type == OsmSharp.OsmGeoType.Node || 
    (osmGeo.Type == OsmSharp.OsmGeoType.Way && osmGeo.Tags != null && osmGeo.Tags.Contains("power", "line"))
   select osmGeo;

  // convert to complete stream.
  var complete = filtered.ToComplete();
}

To extract geographical information from OSM data (like PBF files) and convert it to a Shapefile, you should follow a streaming pipeline:

1. Create a source stream: Use PBFOsmStreamSource to read the OSM file.
2. Filter OSM objects: Use LINQ to filter the OsmGeo objects (e.g., filtering by OsmGeoType or specific tags) before converting to a feature stream. This is critical for memory management; converting all objects to a feature stream will keep them in-memory.
3. Convert to FeatureSource: Use the .ToFeatureSource() extension method on your filtered OSM stream.
4. Filter Geometries: Filter the resulting features by their geometry type (e.g., where feature.Geometry is LineString).
5. Build a FeatureCollection: Create a FeatureCollection and an AttributesTable. When adding features, ensure you use a constant number of attributes with the same names for all features to maintain Shapefile compatibility.
6. Write to Shapefile: Use ShapefileDataWriter.GetHeader to generate a header based on the first feature and the total count, then use ShapefileDataWriter.Write to save the collection.

using (var fileStream = File.OpenRead("path/to/file.osm.pbf"))
{
  // 1. Create source stream
  var source = new PBFOsmStreamSource(fileStream);
  
  // 2. Filter OSM objects (CRITICAL: filter BEFORE ToFeatureSource to save memory)
  var filtered = from osmGeo in source
   where osmGeo.Type == OsmSharp.OsmGeoType.Node || 
    (osmGeo.Type == OsmSharp.OsmGeoType.Way && osmGeo.Tags != null && osmGeo.Tags.Contains("power", "line"))
   select osmGeo;

  // 3. Convert to feature stream
  var features = filtered.ToFeatureSource();

  // 4. Filter geometries
  var lineStrings = from feature in features
    where feature.Geometry is LineString
    select feature;

 // 5. Build feature collection
 var featureCollection = new FeatureCollection();
 var attributesTable = new AttributesTable();
 attributesTable.AddAttribute("type", "powerline");
 foreach (var feature in lineStrings)
 {
    // Ensure constant # of attributes with same names before writing
    featureCollection.Add(new Feature(feature.Geometry, attributesTable));
 }

 // 6. Convert to shape
 var header = ShapefileDataWriter.GetHeader(featureCollection.Features.First(), featureCollection.Features.Count);
 var shapeWriter = new ShapefileDataWriter("luxembourg.shp", new GeometryFactory())
 {
    Header = header
 };
 shapeWriter.Write(featureCollection.Features);
}