Enhance the way of linear search

Question

Enhance the way of linear search

Jonathan 170

Hi,

To C# linear search below

User's image

any advice to apply the following way on the above?

Jonathan 170 Reputation points

2026-06-01T02:17:21.81+00:00

Sorry to all, the screen below was missing (from the above post). How to apply the way below by adjusting the original code?
Bruce (SqlWork.com) 84,086 Reputation points

2026-06-01T14:25:21.7566667+00:00

It’s not clear how this code is related to the first code.

As the file read is the most expensive part, adding parallel processing to scan one line of text scan and blocking on the add to collection, the routine is probably slower than using a regular collection and a no parallel processing. That is the overhead of creating and managing threads is higher than a string scan. Also the blocking for concurrency is slower than a standard collection.

Answer accepted by question author

Nancy Vo (WICLOUD CORPORATION) 5,700 Microsoft External Staff Moderator

Hi @Jonathan ,

I apologize for the delayed response. I have received your sharing regarding the issue.

The core issue is that your current code reads the entire file on every single IP check. This means if 1,000 visitors arrive at the same time, your program opens and reads the file 1,000 times.

I have already tested this on my side and checked the performance.

You can refer to my example code below:

BlockIPManager.cs:

public static class BlockedIPManager
{
    private static readonly object _lock = new object();
    private static HashSet<string> _blockedIPs = null;
    private static readonly string FileName = @"c:\cmp\list11.txt";

    public static void Initialize()
    {
        lock (_lock)
        {
            _blockedIPs = File.Exists(FileName)
                ? File.ReadLines(FileName)
                      .Select(line => line.Trim())
                      .Where(line => !string.IsNullOrWhiteSpace(line))
                      .ToHashSet()
                : new HashSet<string>();
        }
    }

    public static bool ShouldBlockIP(string ip)
    {
        if (string.IsNullOrWhiteSpace(ip)) return false;
        ip = ip.Trim();

        lock (_lock)
        {
            if (_blockedIPs == null) Initialize();

            if (_blockedIPs.Contains(ip))
                return true;

            _blockedIPs.Add(ip);
            File.AppendAllText(FileName, ip + Environment.NewLine);
            return false;
        }
    }

    public static void AddBlockIPs(IEnumerable<string> ipList)
    {
        if (ipList == null) return;

        lock (_lock)
        {
            if (_blockedIPs == null) Initialize();

            var newIPs = ipList
                .Select(ip => ip?.Trim())
                .Where(ip => !string.IsNullOrWhiteSpace(ip)
                          && !_blockedIPs.Contains(ip))
                .Distinct()
                .ToList();

            if (newIPs.Count == 0) return;

            foreach (var ip in newIPs)
                _blockedIPs.Add(ip);

            File.AppendAllLines(FileName, newIPs);
        }
    }
}

Program.cs

using System.Diagnostics;

BlockedIPManager.Initialize();

var sw = Stopwatch.StartNew();

Console.WriteLine("TEST 1: Single IP lookup — IP at TOP of file (10,000 times)");
sw.Restart();
for (int i = 0; i < 10000; i++)
    BlockedIPManager.ShouldBlockIP("113.45.67.89");
sw.Stop();
Console.WriteLine("Result: " + sw.ElapsedMilliseconds + " ms");
Console.WriteLine("Average per lookup: " + (sw.Elapsed.TotalMicroseconds / 10000).ToString("F4") + " us");

Console.WriteLine("");
Console.WriteLine("TEST 2: Single IP lookup — IP at BOTTOM of file (10,000 times)");
sw.Restart();
for (int i = 0; i < 10000; i++)
    BlockedIPManager.ShouldBlockIP("192.168.1.255");
sw.Stop();
Console.WriteLine("Result: " + sw.ElapsedMilliseconds + " ms");
Console.WriteLine("Average per lookup: " + (sw.Elapsed.TotalMicroseconds / 10000).ToString("F4") + " us");

Console.WriteLine("");
Console.WriteLine("TEST 3: Bulk add 1,000 new IPs");
var bulkIPs = Enumerable.Range(1, 1000)
    .Select(i => $"10.10.{i / 256}.{i % 256}")
    .ToArray();
sw.Restart();
BlockedIPManager.AddBlockIPs(bulkIPs);
sw.Stop();
Console.WriteLine("Result: " + sw.ElapsedMilliseconds + " ms");

Console.WriteLine("");
Console.WriteLine("TEST 4: Duplicate Protection");
string testIP = "77.77.77.77";
BlockedIPManager.ShouldBlockIP(testIP);
BlockedIPManager.ShouldBlockIP(testIP);
BlockedIPManager.ShouldBlockIP(testIP);
var count = File.ReadAllLines(@"c:\cmp\list11.txt")
    .Count(l => l.Trim() == testIP);
Console.WriteLine(testIP + " appears in file: " + count + " time(s) (should be 1)");

Console.WriteLine("");
Console.WriteLine("TEST 5: Functional Check");
bool result1 = BlockedIPManager.ShouldBlockIP("123.45.67.89");
Console.WriteLine("TEST 5a - Existing IP blocked: " + result1);

bool result2 = BlockedIPManager.ShouldBlockIP("8.8.8.8");
Console.WriteLine("TEST 5b - New IP added: " + !result2);

bool result3 = BlockedIPManager.ShouldBlockIP("8.8.8.8");
Console.WriteLine("TEST 5c - Same IP now blocked: " + result3);

Console.ReadLine();

User's image

I hope you find this information helpful. Thank you.

Jonathan 170 Reputation points

2026-06-05T03:20:03.5666667+00:00

Billions thanks to Nancy. When did you call AddBlockIPs? My main process is like

Nancy Vo (WICLOUD CORPORATION) 5,700 Microsoft External Staff Moderator

Hi @Jonathan ,

I added AddBlockIPs for testing purposes. In your case, you don’t need to use it. I recommend calling Initialize() once at startup and using ShouldBlockIP() inside your loop.

public static class BlockedIPManager
{
    private static readonly object _lock = new object();
    private static HashSet<string> _blockedIPs = null;
    private static readonly string FileName = @"c:\cmp\list11.txt";

    public static void Initialize()
    {
        lock (_lock)
        {
            _blockedIPs = File.Exists(FileName)
                ? File.ReadLines(FileName)
                      .Select(line => line.Trim())
                      .Where(line => !string.IsNullOrWhiteSpace(line))
                      .ToHashSet()
                : new HashSet<string>();
        }
    }

    public static bool ShouldBlockIP(string ip)
    {
        if (string.IsNullOrWhiteSpace(ip)) return false;
        ip = ip.Trim();

        lock (_lock)
        {
            if (_blockedIPs == null) Initialize();

            if (_blockedIPs.Contains(ip))
                return true;

            _blockedIPs.Add(ip);
            File.AppendAllText(FileName, ip + Environment.NewLine);
            return false;
        }
    }
}

In your main process:

string[] fileEntries = Directory.GetFiles(inputPath);

BlockedIPManager.Initialize();

foreach (string fileName in fileEntries)
{
    using (StreamWriter sw = new StreamWriter(outputPath, true, Encoding.Default))
    {
        foreach (string line in File.ReadLines(fileName))
        {
            if (string.IsNullOrWhiteSpace(line)) continue;

            LogEntry entry = LogEntry.Parse(line);

            if (entry == null) continue;

            if (entry.Message.Contains("53") ||
                entry.Message.Contains("TL") ||
                entry.Message.Contains("re"))
            {
                string ip = entry.GetIPAddress();

                if (!string.IsNullOrEmpty(ip) && !BlockedIPManager.ShouldBlockIP(ip))
                {
                    string ruleName = $"Rule Server...";
                    string firewallCommand = $"netsh...rule name=\"{ip}\"...";

                    Console.WriteLine($"Blocked {ip} | Reason: {entry.Message}");
                    sw.WriteLine(firewallCommand);
                }
            }
        }
    }
}

This way the file is read only once regardless of how many log files or lines you process. All IP lookups after that happen in memory instantly.

Hope this clears things up.

Jonathan 170 Reputation points

2026-06-05T10:16:26.98+00:00

Appreciated a lot to your sincere help! Any advice to my main loop below?
Nancy Vo (WICLOUD CORPORATION) 5,700 Reputation points Microsoft External Staff Moderator

2026-06-05T10:49:22.01+00:00

Hi @Jonathan ,

I actually already included the updated main loop in my previous response. I wonder if you want any other changes or something? If yes, please tell me detailed purpose and I'd glad to support.
Jonathan 170 Reputation points

2026-06-06T06:56:55.8333333+00:00

Good day,

Did you create your static class as a new one? How to put that to the one below?

Nancy Vo (WICLOUD CORPORATION) 5,700 Microsoft External Staff Moderator

Yes @Jonathan ,

I create a new file and name it BlockedIPManager, you can replace:

internal class Class1
{
}

with:

public static class BlockedIPManager
{
    private static readonly object _lock = new object();
    private static HashSet<string> _blockedIPs = null;
    private static readonly string FileName = @"c:\cmp\list11.txt";

    public static void Initialize()
    {
        lock (_lock)
        {
            _blockedIPs = File.Exists(FileName)
                ? File.ReadLines(FileName)
                      .Select(line => line.Trim())
                      .Where(line => !string.IsNullOrWhiteSpace(line))
                      .ToHashSet()
                : new HashSet<string>();
        }
    }

    public static bool ShouldBlockIP(string ip)
    {
        if (string.IsNullOrWhiteSpace(ip)) return false;
        ip = ip.Trim();

        lock (_lock)
        {
            if (_blockedIPs == null) Initialize();

            if (_blockedIPs.Contains(ip))
                return true;

            _blockedIPs.Add(ip);
            File.AppendAllText(FileName, ip + Environment.NewLine);
            return false;
        }
    }
}

Jonathan 170 Reputation points

2026-06-08T03:51:44.17+00:00

I did it. I named that new file as "BlockedIPManager" (green below). Any idea to the red error?
Nancy Vo (WICLOUD CORPORATION) 5,700 Reputation points Microsoft External Staff Moderator

2026-06-08T03:59:32.2966667+00:00

Hi @Jonathan ,

Please share the full BlockIPManager.cs and Program.cs files. Since all class files need to use the same namespace to access each other, I’d like to check whether they are consistent. Thank you.
Jonathan 170 Reputation points

2026-06-08T04:22:06.28+00:00

FYI
Nancy Vo (WICLOUD CORPORATION) 5,700 Reputation points Microsoft External Staff Moderator

2026-06-08T04:30:35.4733333+00:00

Hi @Jonathan ,

The error is caused by a namespace mismatch. Your Program.cs uses namespace LogScanner but BlockedIPManager.cs uses namespace ScanFolder. They need to match

Also, I noticed your Program.cs still has the old ShouldBlockIP method inside the Program class. Please delete that old method entirely - it is no longer needed since BlockedIPManager now handles everything.
Jonathan 170 Reputation points

2026-06-08T05:03:07.5533333+00:00

Original issue was resolved. It is one side issue regarding the current problem to write one unique IP on that. You can tried it furter.

Since I ran the current codes and still had the same problem.

You can also use the 2 events below as the whole one.
Nancy Vo (WICLOUD CORPORATION) 5,700 Reputation points Microsoft External Staff Moderator

2026-06-08T05:58:55.36+00:00

Thank you for your feedback.
Jonathan 170 Reputation points

2026-06-08T07:32:09.86+00:00

It is the same issue to write to list11.txt properly. Can you have a look further? Here is what I can see, at the end of list11.txt, after running your current codes?

Could you have the testing files having the line like it below, to verify the codes? You can put the different IPs

"SMTPD" 3684 4960 "2026-06-08 04:28:13.033" "219.257.337.283" "SENT: 535 Authentication failed. Restarting authentication process."

Nancy Vo (WICLOUD CORPORATION) 5,700 Microsoft External Staff Moderator

Hi @Jonathan ,

I do not see any additional lines at the end of list11.txt.

I created a new file named test.log.txt:

"SMTPD" 3684 4960 "2026-06-08 04:28:13.033" "219.257.337.283" "SENT: 535 Authentication failed. Restarting authentication process."
"SMTPD" 3684 4960 "2026-06-08 04:28:13.033" "8.8.8.8" "SENT: 535 Authentication failed. Restarting authentication process."
"SMTPD" 3684 4960 "2026-06-08 04:28:13.033" "1.2.3.4" "SENT: 535 Authentication failed. Restarting authentication process."
"SMTPD" 3684 4960 "2026-06-08 04:28:13.033" "219.257.337.283" "SENT: 535 Authentication failed. Restarting authentication process."

and this is result:

User's image

For further testing, I adjusted the code. You can refer to my example below:

BlockedIPManager.cs:

using System.Text.RegularExpressions;

public static class BlockedIPManager
{
    private static readonly object _lock = new object();
    private static HashSet<string> _blockedIPs = null;
    private static readonly string FileName = @"c:\cmp\list11.txt";

    private static readonly Regex IpRegex = new Regex(
        @"\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b", RegexOptions.Compiled);

    public static void Initialize()
    {
        lock (_lock)
        {
            LoadAndCleanFile();
        }
    }

    private static void LoadAndCleanFile()
    {
        if (!File.Exists(FileName))
        {
            _blockedIPs = new HashSet<string>();
            return;
        }

        var validIPs = new HashSet<string>();
        var validLines = new List<string>();

        foreach (var line in File.ReadLines(FileName))
        {
            string ip = ExtractIP(line);
            if (!string.IsNullOrEmpty(ip) && validIPs.Add(ip))
            {
                validLines.Add(ip);
            }
        }

        File.WriteAllLines(FileName, validLines);

        _blockedIPs = validIPs;
        Console.WriteLine($"[INFO] Loaded and cleaned {_blockedIPs.Count} valid IPs from file.");
    }

    private static string ExtractIP(string text)
    {
        if (string.IsNullOrWhiteSpace(text)) return null;
        var match = IpRegex.Match(text);
        return match.Success ? match.Groups[1].Value : null;
    }

    public static bool ShouldBlockIP(string input)
    {
        if (string.IsNullOrWhiteSpace(input)) return false;

        string ip = ExtractIP(input);
        if (string.IsNullOrEmpty(ip)) return false;

        lock (_lock)
        {
            if (_blockedIPs == null) Initialize();

            if (_blockedIPs.Contains(ip))
                return true;

            _blockedIPs.Add(ip);
            File.AppendAllText(FileName, ip + Environment.NewLine);
            return false;
        }
    }

    public static void AddBlockIPs(IEnumerable<string> ipList)
    {
        if (ipList == null || !ipList.Any()) return;

        lock (_lock)
        {
            if (_blockedIPs == null) Initialize();

            var newIPs = ipList
                .Select(ExtractIP)
                .Where(ip => !string.IsNullOrEmpty(ip) && !_blockedIPs.Contains(ip))
                .Distinct()
                .ToList();

            if (newIPs.Count == 0) return;

            foreach (var ip in newIPs)
                _blockedIPs.Add(ip);

            File.AppendAllLines(FileName, newIPs);
        }
    }
}

Program.cs:

using System.Diagnostics;

BlockedIPManager.Initialize();

var sw = Stopwatch.StartNew();

Console.WriteLine("TEST 1: Single IP lookup — IP at TOP of file (10,000 times)");
sw.Restart();
for (int i = 0; i < 10000; i++)
    BlockedIPManager.ShouldBlockIP("113.45.67.89");
sw.Stop();
Console.WriteLine("Result: " + sw.ElapsedMilliseconds + " ms");
Console.WriteLine("Average per lookup: " + (sw.Elapsed.TotalMicroseconds / 10000).ToString("F4") + " us");

Console.WriteLine("");
Console.WriteLine("TEST 2: Single IP lookup — IP at BOTTOM of file (10,000 times)");
sw.Restart();
for (int i = 0; i < 10000; i++)
    BlockedIPManager.ShouldBlockIP("192.168.1.255");
sw.Stop();
Console.WriteLine("Result: " + sw.ElapsedMilliseconds + " ms");
Console.WriteLine("Average per lookup: " + (sw.Elapsed.TotalMicroseconds / 10000).ToString("F4") + " us");

Console.WriteLine("");
Console.WriteLine("TEST 3: Bulk add 1,000 new IPs");
var bulkIPs = Enumerable.Range(1, 1000)
    .Select(i => $"10.10.{i / 256}.{i % 256}")
    .ToArray();
sw.Restart();
BlockedIPManager.AddBlockIPs(bulkIPs);
sw.Stop();
Console.WriteLine("Result: " + sw.ElapsedMilliseconds + " ms");

Console.WriteLine("");
Console.WriteLine("TEST 4: Duplicate Protection");
string testIP = "77.77.77.77";
BlockedIPManager.ShouldBlockIP(testIP);
BlockedIPManager.ShouldBlockIP(testIP);
BlockedIPManager.ShouldBlockIP(testIP);
var count = File.ReadAllLines(@"c:\cmp\list11.txt")
    .Count(l => l.Trim() == testIP);
Console.WriteLine(testIP + " appears in file: " + count + " time(s) (should be 1)");

Console.WriteLine("");
Console.WriteLine("TEST 5: Functional Check");
bool result1 = BlockedIPManager.ShouldBlockIP("123.45.67.89");
Console.WriteLine("TEST 5a - Existing IP blocked: " + result1);

bool result2 = BlockedIPManager.ShouldBlockIP("8.8.8.8");
Console.WriteLine("TEST 5b - New IP added: " + !result2);

bool result3 = BlockedIPManager.ShouldBlockIP("8.8.8.8");
Console.WriteLine("TEST 5c - Same IP now blocked: " + result3);

Console.WriteLine("\n=== test.log.txt ===");

string[] testLogLines = File.ReadAllLines(@"c:\cmp\test.log.txt");

foreach (var logLine in testLogLines)
{
    bool isBlocked = BlockedIPManager.ShouldBlockIP(logLine);
    Console.WriteLine($"Processed line → Blocked: {isBlocked}");
}

Console.WriteLine("\nFinal content of list11.txt (last 15 lines):");
var finalLines = File.ReadAllLines(@"c:\cmp\list11.txt")
                     .TakeLast(15)
                     .ToArray();

foreach (var line in finalLines)
{
    Console.WriteLine("→ " + line);
}

Console.ReadLine();

Hope this points in the right direction.

Jonathan 170 Reputation points

2026-06-09T15:47:30.2+00:00

Appreciated a lot to your help! I tried to make use of your current class posted above. After having run that, it is having long time to open list11.txt file and there should be still the unexpected Unicode on that.

Any other advice that I could share more of my project to verify that? Supposing that we're doing the correct thing matching the code of conduct on this forum.

Really many thanks to your help!

Bruce (SqlWork.com) 84,086

your file append is not checking the encoding of the existing file, so the default utf8 append is not working:

try:

public static Encoding GetFileEncoding(string filePath)
{
    // Fallback default if no BOM is found
    Encoding detectedEncoding = Encoding.UTF8; 

    if (File.Exists(filePath) 
    {
       // Read the BOM to identify the encoding
       using (var reader = new StreamReader(filePath, Encoding.UTF8, true))
       {
           if (reader.Peek() >= 0) // Peak forces the stream to read and check BOM
           {
               reader.Read(); 
           }
           detectedEncoding = reader.CurrentEncoding;
       }
    }
    return detectedEncoding;
}

then use on append:

   var encoding = GetFileEncoding(fileName);
   File.AppendAllLines(FileName, newIPs, encoding)

Jonathan 170 Reputation points

2026-06-10T04:47:58.14+00:00

Bruce,

You're correct. Thanks a lot! Appreciated!

Nancy/Bruce,

One side question, to that Hashset line, any way to detect the current IP is not existing within set/list, and to further add that if not existing. Billions of thanks to both of you!
Nancy Vo (WICLOUD CORPORATION) 5,700 Reputation points Microsoft External Staff Moderator

2026-06-12T07:06:15.0533333+00:00
Hi @Jonathan

To detect whether the current IP does not exist within a set/list, you can use Contains(). Please refer to the example code below:

if (!_blockedIPs.Contains(ip)) { _blockedIPs.Add(ip); File.AppendAllLines(FileName, new[] { ip }, GetFileEncoding(FileName)); }

Thank you.

4 additional answers

Your answer

Jonathan 170 Reputation points

2026-06-01T02:17:21.81+00:00

Sorry to all, the screen below was missing (from the above post). How to apply the way below by adjusting the original code?
Bruce (SqlWork.com) 84,086 Reputation points

2026-06-01T14:25:21.7566667+00:00

It’s not clear how this code is related to the first code.

As the file read is the most expensive part, adding parallel processing to scan one line of text scan and blocking on the add to collection, the routine is probably slower than using a regular collection and a no parallel processing. That is the overhead of creating and managing threads is higher than a string scan. Also the blocking for concurrency is slower than a standard collection.
Jonathan 170 Reputation points

2026-06-05T03:20:03.5666667+00:00

Billions thanks to Nancy. When did you call AddBlockIPs? My main process is like
Jonathan 170 Reputation points

2026-06-05T10:16:26.98+00:00

Appreciated a lot to your sincere help! Any advice to my main loop below?
Nancy Vo (WICLOUD CORPORATION) 5,700 Reputation points Microsoft External Staff Moderator

2026-06-05T10:49:22.01+00:00

Hi @Jonathan ,

I actually already included the updated main loop in my previous response. I wonder if you want any other changes or something? If yes, please tell me detailed purpose and I'd glad to support.
Jonathan 170 Reputation points

2026-06-06T06:56:55.8333333+00:00

Good day,

Did you create your static class as a new one? How to put that to the one below?
Nancy Vo (WICLOUD CORPORATION) 5,700 Reputation points Microsoft External Staff Moderator

2026-06-08T03:22:53.2633333+00:00

Yes @Jonathan ,

I create a new file and name it BlockedIPManager, you can replace:

internal class Class1 { }

with:

public static class BlockedIPManager { private static readonly object _lock = new object(); private static HashSet<string> _blockedIPs = null; private static readonly string FileName = @"c:\cmp\list11.txt"; public static void Initialize() { lock (_lock) { _blockedIPs = File.Exists(FileName) ? File.ReadLines(FileName) .Select(line => line.Trim()) .Where(line => !string.IsNullOrWhiteSpace(line)) .ToHashSet() : new HashSet<string>(); } } public static bool ShouldBlockIP(string ip) { if (string.IsNullOrWhiteSpace(ip)) return false; ip = ip.Trim(); lock (_lock) { if (_blockedIPs == null) Initialize(); if (_blockedIPs.Contains(ip)) return true; _blockedIPs.Add(ip); File.AppendAllText(FileName, ip + Environment.NewLine); return false; } } }
Jonathan 170 Reputation points

2026-06-08T03:51:44.17+00:00

I did it. I named that new file as "BlockedIPManager" (green below). Any idea to the red error?
Nancy Vo (WICLOUD CORPORATION) 5,700 Reputation points Microsoft External Staff Moderator

2026-06-08T03:59:32.2966667+00:00

Hi @Jonathan ,

Please share the full BlockIPManager.cs and Program.cs files. Since all class files need to use the same namespace to access each other, I’d like to check whether they are consistent. Thank you.
Jonathan 170 Reputation points

2026-06-08T04:22:06.28+00:00

FYI
Nancy Vo (WICLOUD CORPORATION) 5,700 Reputation points Microsoft External Staff Moderator

2026-06-08T04:30:35.4733333+00:00

Hi @Jonathan ,

The error is caused by a namespace mismatch. Your Program.cs uses namespace LogScanner but BlockedIPManager.cs uses namespace ScanFolder. They need to match

Also, I noticed your Program.cs still has the old ShouldBlockIP method inside the Program class. Please delete that old method entirely - it is no longer needed since BlockedIPManager now handles everything.
Jonathan 170 Reputation points

2026-06-08T05:03:07.5533333+00:00

Original issue was resolved. It is one side issue regarding the current problem to write one unique IP on that. You can tried it furter.

Since I ran the current codes and still had the same problem.

You can also use the 2 events below as the whole one.
Nancy Vo (WICLOUD CORPORATION) 5,700 Reputation points Microsoft External Staff Moderator

2026-06-08T05:58:55.36+00:00

Thank you for your feedback.
Jonathan 170 Reputation points

2026-06-08T07:32:09.86+00:00

It is the same issue to write to list11.txt properly. Can you have a look further? Here is what I can see, at the end of list11.txt, after running your current codes?

Could you have the testing files having the line like it below, to verify the codes? You can put the different IPs

"SMTPD" 3684 4960 "2026-06-08 04:28:13.033" "219.257.337.283" "SENT: 535 Authentication failed. Restarting authentication process."
Jonathan 170 Reputation points

2026-06-09T15:47:30.2+00:00

Appreciated a lot to your help! I tried to make use of your current class posted above. After having run that, it is having long time to open list11.txt file and there should be still the unexpected Unicode on that.

Any other advice that I could share more of my project to verify that? Supposing that we're doing the correct thing matching the code of conduct on this forum.

Really many thanks to your help!
Bruce (SqlWork.com) 84,086 Reputation points

2026-06-09T19:02:30.2533333+00:00

your file append is not checking the encoding of the existing file, so the default utf8 append is not working:

try:

public static Encoding GetFileEncoding(string filePath) { // Fallback default if no BOM is found Encoding detectedEncoding = Encoding.UTF8; if (File.Exists(filePath) { // Read the BOM to identify the encoding using (var reader = new StreamReader(filePath, Encoding.UTF8, true)) { if (reader.Peek() >= 0) // Peak forces the stream to read and check BOM { reader.Read(); } detectedEncoding = reader.CurrentEncoding; } } return detectedEncoding; }

then use on append:

var encoding = GetFileEncoding(fileName); File.AppendAllLines(FileName, newIPs, encoding)
Jonathan 170 Reputation points

2026-06-10T04:47:58.14+00:00

Bruce,

You're correct. Thanks a lot! Appreciated!

Nancy/Bruce,

One side question, to that Hashset line, any way to detect the current IP is not existing within set/list, and to further add that if not existing. Billions of thanks to both of you!
Nancy Vo (WICLOUD CORPORATION) 5,700 Reputation points Microsoft External Staff Moderator

2026-06-12T07:06:15.0533333+00:00

Hi @Jonathan

To detect whether the current IP does not exist within a set/list, you can use Contains(). Please refer to the example code below:

if (!_blockedIPs.Contains(ip)) { _blockedIPs.Add(ip); File.AppendAllLines(FileName, new[] { ip }, GetFileEncoding(FileName)); }

Thank you.

Answer 1

Nancy Vo (WICLOUD CORPORATION) 5,700 Microsoft External Staff Moderator

Hello @Jonathan ,

Thanks for your question.

Instead of applying SearchValues + Parallel.ForEach to your existing List, I would suggest a simpler and more effective approach - replace List<string> with HashSet<string>.

You can refer to my example code:

var ipSet = new HashSet<string>(
    File.Exists(@"c:/cmp/list11.txt")
        ? File.ReadLines(@"c:/cmp/list11.txt").Select(l => l.Trim())
        : Enumerable.Empty<string>(),
    StringComparer.OrdinalIgnoreCase
);

public static bool AddIfNotExists(string ip, HashSet<string> ipSet)
{
    ip = ip.Trim();

    if (ipSet.Add(ip))
    {
        File.AppendAllText(@"c:/cmp/list11.txt", ip + Environment.NewLine);
        return true;
    }

    return false;
}

I hope this addresses your question. If this response was helpful, please consider following the guidance to provide feedback.

Jonathan 170 Reputation points

2026-06-01T08:07:27.1166667+00:00

Thanks a lot to Nancy for your code. From IpSet, which boolean or value should I make use of, to decide it for the original line below?

ind0 = list0.FindIndex(item => item == ip);
Nancy Vo (WICLOUD CORPORATION) 5,700 Reputation points Microsoft External Staff Moderator

2026-06-01T08:23:37.7166667+00:00
Hi @Jonathan ,

Please replace the following code:

ind0 = list0.FindIndex(item => item == ip); if (ind0 < 0) { list0.Add(ip); // ... write file return true; }

with:

if (ipSet.Add(ip)) { // ... write file return true; }
Jonathan 170 Reputation points

2026-06-01T08:28:18.24+00:00

When did you refer to AddIfNotExists?
Nancy Vo (WICLOUD CORPORATION) 5,700 Reputation points Microsoft External Staff Moderator

2026-06-01T08:33:33.64+00:00

AddIfNotExists is the same method you already have, just rewritten. Therefore, you only need to replace the body of your existing method with my example code - there’s no need to create a new method named AddIfNotExists.
Jonathan 170 Reputation points

2026-06-01T09:48:17.5866667+00:00

Sorry, when did you see AddIfNotExists event was existing on my original code?
Jonathan 170 Reputation points

2026-06-02T03:58:31.3566667+00:00

Sorry, how to adjust it below?

Nancy Vo (WICLOUD CORPORATION) 5,700 Microsoft External Staff Moderator

Hi @Jonathan ,

Please refer to my example code and let me know the results. I’d be happy to investigate further if needed.

static bool ShouldBlockIP(string ip)
{
    if (string.IsNullOrEmpty(ip)) return false;

    string[] safePrefixes = { "113.", "223.", "127.", "103.15", "182.173", "40.107", "142.251", "142.250" };

    var ipSet = new HashSet<string>(
        File.Exists(@"c:/cmp/list11.txt")
            ? File.ReadLines(@"c:/cmp/list11.txt").Select(l => l.Trim())
            : Enumerable.Empty<string>(),
        StringComparer.OrdinalIgnoreCase
    );

    if (ipSet.Add(ip))
    {
        File.AppendAllText(@"c:/cmp/list11.txt", ip + Environment.NewLine);
        return true;
    }

    return false;
}

Jonathan 170 Reputation points

2026-06-02T09:21:54.22+00:00

Many thanks to your nice help! Is it still a little bit of issue on this since I got repeated IP like

let me know if you want to see more details on my project.
Nancy Vo (WICLOUD CORPORATION) 5,700 Reputation points Microsoft External Staff Moderator

2026-06-02T09:31:48.27+00:00

Thanks for sharing. Could you help me better understand the purpose of ShouldBlockIP? I just want to make sure I understand correctly - please clarify your intended use case so I can investigate further.
Jonathan 170 Reputation points

2026-06-02T09:43:47.4033333+00:00

List11 file is having all Unique IPs ever blocked. I saw the file is having this

unexpectedly at the end of the file. It was having the issue to write to the file correctly!
Jonathan 170 Reputation points

2026-06-04T02:04:11.9466667+00:00

Nancy, Do you need more details for this? There was the issue to write one IP properly into the text file for your way. Could you see this? Or can anyone help?

Bruce (SqlWork.com) 84,086

as the question was about performance, the hash set is lowering it. as the ip test causes a file I/O, and the hash set is only used once it is slower than just a compare.. worse, you always read the whole file, even if there is a match in first row. try:

static bool ShouldBlockIP(string ip)
{
    var fileName = @"c:\cmp\list11.txt";
    ip = ip?.Trim()

    if (string.IsNullOrEmpty(ip)) return false;

    string[] safePrefixes = { "113.", "223.", "127.", "103.15", "182.173", "40.107", "142.251", "142.250" };

    // test if ip in file contents
    // as ip's are numbers, no need for case insensitive compare
    var found = File.Exists(fileName)
            ? File.ReadLines(fileName).FindIndex(l => ip == l.Trim()) >= 0
            : false;

    // if ip not found, append to text file with line ending
    if (!found)
    {
        File.AppendAllText(fileName, ip + Environment.NewLine);
    }

    return found;
}

if you passed a list of ipaddress, then you could limit I/O (where all the time is). and use hash sets.

static bool AddBlockIPs(string[] ipList)
{
    var fileName = @"c:\cmp\list11.txt";
    var safePrefixes = new HashSet<string> { "113.", "223.", "127.", "103.15", "182.173", "40.107", "142.251", "142.250" };

    // convert file contents to hashset
    var fileHashSet = File.Exists(fileName)
            ? File.ReadLines(fileName)
                  .Select(l => l.Trim())
                  .ToHashSet()                  
            : new HashSet<string>();

    // for each ip not in file hash add to newRows
    // optional filter for prefixes
    var newRowsText = string.Join("\n",
         (ipList ?? [])
           .Select(r => r.Trim())
           /* .Where(r => 
			{
				var prefix = r.Split(".")[0] + "."; 
				return !safePrefixes.Contains(prefix);
			}) */
            .Where(r => !FileHashSet.Contains(r))
         );
  
    if (newRows.Length > 0)
    {
        File.AppendAllText(fileName, newRowsText);
        return true
    }

    return false;
}

Jonathan 170 Reputation points

2026-06-04T03:50:11.7966667+00:00

Sorry to the issues below for your codes.

Answer 2

Bruce (SqlWork.com) 84,086

The file I/O is orders of magnitude slower than the index lookup, you will never notice the difference. If the lookup is in a loop, you want to extract the file access to outside the loop. That is build a new list, the write the list once. also you don’t show how lookups are done. Converting the list to a dictionary or hash requires iteration over the list, so if there is only one lookup, the conversion is more expensive. Also if the list is small, find index is not expensive, and may be quicker.

0 comments

Answer 3

Viorel 127K

The solution depends on other details. For example, if the list is loaded from file, then:

string path = @"c:\cmp\list11.txt";

ip = ip.Trim( );

if( !File.Exists( path ) || !File.ReadLines( path ).Any( a => a == ip ) )
{
    File.AppendAllText( path, ip + Environment.NewLine );

    return true;
}

0 comments

Answer 4

List<T>.FindIndex performs a linear search (O(n)) on every lookup. For frequent lookups or large collections, replace the list-based search with a data structure that provides O(1) lookup.

A typical enhancement is to use HashSet<string> or Dictionary<string, TValue> instead of repeatedly calling FindIndex on List<string>:

// Build a HashSet once from the existing list
var ipSet = new HashSet<string>(list0);  // case-sensitive; use custom comparer if needed

// When checking/adding a new IP
if (!ipSet.Contains(ip))
{
    ipSet.Add(ip);
}

// When persisting back to file, enumerate the set
File.Delete(@"c:/cmp/list11.txt");

using (var sw = new StreamWriter(@"c:/cmp/list11.txt", append: false))
{
    foreach (var item in ipSet)
    {
        sw.WriteLine(item.Trim());
    }
}

Key improvements compared to the original linear search with FindIndex:

HashSet<string>.Contains is O(1) on average, instead of O(n) for List<T>.FindIndex.
The output file is opened once and written in a single pass, instead of opening a new StreamWriter inside the loop.
The collection (ipSet) naturally enforces uniqueness, so duplicates are avoided without extra checks.

If the order of elements must be preserved, maintain both a List<string> for ordering and a HashSet<string> for fast membership checks:

var ipList = new List<string>();
var ipSet  = new HashSet<string>();

void AddIp(string ip)
{
    if (ipSet.Add(ip))
    {
        ipList.Add(ip); // only added when new
    }
}

This pattern keeps lookups efficient while preserving insertion order for file output.

References:

Share via

Enhance the way of linear search

4 additional answers

Your answer