OpenStreetMap Access blocked

[edlestau]

OpenStreetMap is blocked as described here

https://wiki.openstreetmap.org/wiki/Blocked_tiles

[Quax82]

Happened to me as well. I solved it by writing a custom provider and do the download manually as required by the usage policy.

[JohnnyBravo75]

Happens to me as well.

@Quak82
Can you share the code?

[edlestau]

@Quax82 : or publish as a new version here?
Is that possible?

[Quax82]

Sure... I did not create a fork so I can't create the pull request. This code is straight out of my current project and the version has not yet been released but it works for me. It needs optimization though - so if you have any suggestion, let me know.

The issues for me were:

• Incorrect tile URL
• Generic client identification
  so I had to write my own 'fetch' function using HttpClient:

`/***** Namespaces *****/
using System.IO;
using System.Net.Http;
using System.Reflection;

using GMap.NET;
using GMap.NET.MapProviders;
using GMap.NET.WindowsPresentation;

using MyProject.Settings.MyProject;

/***** Namespace *****/
namespace MyProject.Controls.GMap.MapProviders;

#region CustomMapProvider class
internal abstract class CustomMapProvider : GMapProvider, IMapProvider
{
#region Constructors
protected CustomMapProvider(string providerName, string affiliation, IMyProjectSettings settings)
{
/***** Initialize settings *****/
Settings = settings ?? throw new ArgumentNullException(nameof(settings));

	/***** Initialize fields *****/
	_name = providerName ?? throw new ArgumentNullException(nameof(providerName));
	_affiliation = affiliation ?? throw new ArgumentNullException(nameof(affiliation));

	/***** Initialize HTTP client *****/
	// Get MyProject version
	var version = Assembly.GetEntryAssembly()?.GetName().Version?.ToString() ?? "0.0.0.0";

	// Build user agent
	var client = $"MyProject v{version} (http://www.myproject.com - contact: info@myproject.com)";

	// Initialize HttpClient
	Client = new HttpClient();
	Client.DefaultRequestHeaders.Add("User-Agent", client);
}
#endregion

#region GMapProvider implementation
#region Methods
/***** Get the tile image for the specified position and zoom level *****/
public override PureImage? GetTileImage(GPoint pos, int zoom)
{
	try
	{
		/***** Try to load tile from cache *****/
		var tilePath = Path.Combine(Settings.TempPath, $"{Name}__tile_z{zoom}_x{pos.X}_y{pos.Y}.png");
		if (File.Exists(tilePath) && File.GetCreationTimeUtc(tilePath) >= DateTime.UtcNow.AddDays(-CacheExpirationInDays))
		{
			/***** Create and return tile image *****/
			var tileImage = GetTileImageFromFile(tilePath);
			if (tileImage != null)
				return tileImage;
		}

		/***** Load tile from server *****/
		// Build URL
		var url = $"https://tile.openstreetmap.org/{zoom}/{pos.X}/{pos.Y}.png";

		// Fetch tile
		using var responseStream = Client.GetStreamAsync(url).GetAwaiter().GetResult();

		using var ms = new MemoryStream();
		responseStream.CopyTo(ms);
		if (ms.Length == 0)
			return null;

		ms.Position = 0;
		var tile = new GMapImage()
		{
			Data = ms,
		};
		tile.Data.Position = 0;

		/***** Cache tile *****/
		ms.Position = 0;
		if (tile?.Data != null)
		{
			using var tileFileStream = File.Create(tilePath);
			tile.Data.Position = 0;
			tile.Data.CopyTo(tileFileStream);
		}

		/***** Return tile *****/
		return tile is not null ? GetTileImageFromArray(tile.Data.ToArray()) : null;
	}
	catch (Exception ex)
	{
		SentrySdk.CaptureException(ex); ;
		Logger.Error("Failed to load custom tile", ex);
	}

	/***** If anything fails, return empty tile *****/
	return null!;
}
#endregion

#region Properties
public override string Name => _name;
#endregion
#endregion

#region IMapProvider implementation
#region Properties
public string Affiliation
{
	get => _affiliation;
	private init => _affiliation = value;
}
#endregion
#endregion

#region Properties
protected int CacheExpirationInDays => 28;

private HttpClient Client { get; set; }

protected IMyProjectSettings Settings { get; private init; }
#endregion

#region Fields
private string _affiliation;
private string _name;
#endregion

#region Logging
private static readonly log4net.ILog Logger = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);
#endregion

}
#endregion`

Please excuse my coding style - I just like to create sections of code and have comments that are not necessary (e.g. "Namespaces", "Namespace" etc).

Also, the reason why GetTileImageFromArray() is used here

return tile is not null ? GetTileImageFromArray(tile.Data.ToArray()) : null;

is that returning the tile directly resulted in a white/empty tile. I don't know why yet so if anybody has an idea, let me know please.

[edlestau]

Thx a lot

[HikeAndMap]

For those who wish a different approach, I also made another solution.

This solution can also be used to add your own tile server using GeoServer for instance. Hosting your own tiles, in any format that's QGIS compatible and then also can be used in GMap.NET - just adjust the code below

I borrowed some code from Quax82 - no need to reinvent the wheel right, we're all here to help each other and come up with different ideas and solutions
`

public class OpenStreetMapAlt : GMapProvider
{
public static readonly OpenStreetMapAlt Instance;

  OpenStreetMapAlt()
  {
      MaxZoom = 24;
  }

  static OpenStreetMapAlt()
  {
      Instance = new OpenStreetMapAlt();

      Type mytype = typeof(GMapProviders);
      FieldInfo field = mytype.GetField("DbHash", BindingFlags.Static | BindingFlags.NonPublic);
      Dictionary<int, GMapProvider> list = (Dictionary<int, GMapProvider>)field.GetValue(Instance);

      list.Add(Instance.DbId, Instance);
  }

  #region GMapProvider Members

  readonly Guid id = new Guid("4579628D-B532-4CDF-89AE-F20751BBDB2B");

  protected int CacheExpirationInDays => 28;

  public override Guid Id
  {
      get { return id; }
  }

  readonly string name = "OpenStreetMap";

  public override string Name
  {
      get { return name; }
  }

  GMapProvider[] overlays;

  public override GMapProvider[] Overlays
  {
      get
      {
          if (overlays == null)
          {
              overlays = new GMapProvider[] { this };
          }
          return overlays;
      }
  }

  public override PureProjection Projection
  {
      get { return MercatorProjection.Instance; }
  }

  public override PureImage GetTileImage(GPoint pos, int zoom)
  {
      string url = MakeTileImageUrl(pos, zoom, LanguageStr);

      // used cached if exists and not expired
      string path = System.IO.Path.Combine(System.IO.Path.GetTempPath() + "Mapanan\\", $"{Name}__tile_z{zoom}_x{pos.X}_y{pos.Y}.png");
      
      // Make sure the directory exists
      System.IO.FileInfo file = new System.IO.FileInfo(path);
      file.Directory.Create();

      if (System.IO.File.Exists(path) && System.IO.File.GetCreationTimeUtc(path) >= DateTime.UtcNow.AddDays(-CacheExpirationInDays))
      {
          /***** Create and return tile image *****/
          var tileImage = GetTileImageFromFile(path);
          if (tileImage != null)
              return tileImage;
      }

      // Get project version
      var version = Assembly.GetEntryAssembly()?.GetName().Version?.ToString() ?? "0.0.0.0";
      var client = $"Mapanan_Utilities v{version} (http://https://wiki.openstreetmap.org/wiki/Philippines/MAPANAN - contact: vincent@veldman.me)";

      // Initialize HttpClient
      HttpClient Client = new HttpClient();
      Client.DefaultRequestHeaders.Add("User-Agent", client);

      // Fetch tile
      using (var responseStream = Client.GetStreamAsync(url).GetAwaiter().GetResult())
      {
          using (var ms = new MemoryStream())
          {
              responseStream.CopyTo(ms);
              if (ms.Length == 0)
                  return null;

              ms.Position = 0;
              var tile = new GMapImage()
              {
                  Data = ms,
              };
              tile.Data.Position = 0;

              /***** Cache tile *****/
              ms.Position = 0;
              if (tile != null && tile.Data != null)
              {
                  using (var tileFileStream = System.IO.File.Create(path))
                  {
                      tile.Data.Position = 0;
                      tile.Data.CopyTo(tileFileStream);
                  }
              }

              /***** Return tile *****/
              return tile != null ? GetTileImageFromArray(tile.Data.ToArray()) : null;
          }
      }

      /*
      GMapProvider.Credential = new NetworkCredential("Common", "AntonioLunadeSanPedro-05061899");
      ForceBasicHttpAuthentication("Common", "AntonioLunadeSanPedro-05061899");
      GMapProvider.TimeoutMs = 30000;

      try { return GetTileImageUsingHttp(url); } catch (Exception) { return null; }
      */
  }

  #endregion

  string MakeTileImageUrl(GPoint pos, int zoom, string language)
  {
      return $"https://tile.openstreetmap.org/{zoom}/{pos.X}/{pos.Y}.png";
      
  }

}`

And in your main code you add:
`
// basemaps

//listMapProviders.Add(new MapProvider(0, "OpenStreetMap", OpenStreetMapProvider.Instance));
listMapProviders.Add(new MapProvider(0, "OpenStreetMap", OpenStreetMapAlt.Instance));
listMapProviders.Add(new MapProvider(1, "Google Satellite", GoogleSatelliteMapProvider.Instance));
`

Then you disable the build-in OpenStreetMap and replace it with my version.

[RL-Manny]

I have implemented a more simplified approach, as the issue seemed to be that OSM's tile server blocks requests that use a browser-like User-Agent (the fake Firefox string GMap.NET was sending). Their usage policy requires a descriptive User-Agent that identifies your application.

The fix is simply to add an InitializeWebRequest override to OpenStreetMapProviderBase that sets the User-Agent to "GMap.NET" for all OSM-based providers.

Implementation:

Add this field at the top of OpenStreetMapProviderBase:

// OSM tile usage policy requires a descriptive User-Agent identifying the application.
// Using a browser-like User-Agent (e.g. Firefox) causes the server to block requests.
// See: https://operations.osmfoundation.org/policies/tiles/
public static string OsmUserAgent = "GMap.NET";

Then add this override for InitializeWebRequest in OpenStreetMapProviderBase:

protected override void InitializeWebRequest(WebRequest request)
{
        base.InitializeWebRequest(request);
        if (request is System.Net.HttpWebRequest r)
        {
                r.UserAgent = OsmUserAgent;
        }
        else if (!string.IsNullOrEmpty(OsmUserAgent))
        {
                request.Headers["User-Agent"] = OsmUserAgent;
        }
}