Add a static IP alias to a DHCP interface on Windows 8 (and above)

Question

I'm new to Scrapy but so far I managed to code this spider (which actually works as intended, almost).

I want to name the downloaded files f'{issue['number']}_{issue['date']}.pdf' but so far I couldn't. name is just a temporary workaround I came up with.

I was looking at Itemloader documentation, maybe that's what I need, but implies rewriting the whole code. Maybe there's a simpler solution. I will go on reading the documentation.

Any tip is welcome, thanks in advance.

PD: English is my second language, and I'm still reading Scrapy documentation ;) If you are wondering why Venezuela, one of my tasks is to import those documents into our database.

ftp_connection = ftplib.FTP(host=ftp_host, user=ftp_user, passwd=ftp_password)
print(ftp_connection.getwelcome())
ftp_connection.cwd(ftp_directory)
ftp_files = ftp_connection.nlst()
print('Successully created list of file names')
ftp_connection.quit()

class DoVenezuela(scrapy.Spider):

    name = 'do_venezuela'
    start_urls = ['http://spgoin.imprentanacional.gob.ve/cgi-win/be_alex.cgi?forma=FGENERAL&nombrebd=spgoin&c01=Titulo&m01=frase&t01=&c03=Descriptor_TGO1&m03=comienzo&c04=FechaInicio&m04=%3E%3D&t04=01-01-2021&c05=FechaInicio&t05=&c06=Descriptor_EDR1&m06=frase&t06=Publicado&TSalida=T%3AGeneralGCTOF&recuperar=3000&MostrarHijos=E&Cizq=2&xsl=&pxsl=&TipoDoc=GCTOF&Submit2=Buscar&Orden=;FID;']

    def parse(self, response):
        even_rows = response.css('tr.LineaTablaImpar')
        odd_rows = response.css('tr.LineaTablaPar')
        all_rows = even_rows + odd_rows

        for row in all_rows:
            issue = Issue()
            issue['number'] = row.css('a.DocTitulo::text').get().replace('.', '')
            issue['edition'] = row.css('a.RefDescriptor::text').get()
            issue['date'] = row.css('td::text')[2].get().replace('-', '')
            issue['link'] =  row.css('a.DocTitulo').attrib['href']

            lookup = f'{issue["number"]}_{issue["date"]}.pdf'

            if lookup in ftp_files:
                print(f'Skipping {lookup}.pdf: already in ftp')
            elif os.path.isfile(f'{download_directory}/{lookup}'):
                print(f'Skipping {lookup}.pdf: already downloaded')
            else:
                yield response.follow(url=f'http://spgoin.imprentanacional.gob.ve{issue["link"]}', callback=self.parse_link1)

    def parse_link1(self, response):
        link1 = response.css('a')[21].attrib['href']
        yield response.follow(url=f'http://spgoin.imprentanacional.gob.ve{link1}', callback=self.parse_link2)

    def parse_link2(self, response):
        link2 = response.css('a')[17].attrib['href']
        yield response.follow(url=f'http://spgoin.imprentanacional.gob.ve{link2}', callback=self.download_pdf)
    
    def download_pdf(self, response):
        name = response.url.split('/')[4].replace('be_alex.cgi?Documento=', '')
        pdf_file = open(f'{download_directory}/{name}.pdf', 'wb')
        pdf_file.write(response.body)

Solution 1

Finally DHCP and static IPs can be configured to co-exist on one NIC. This feature has landed in the Windows 10 Creators Update (1703).

There is a new property called dhcpstaticipcoexistence in the netsh interface ipv4 set interface command, which can be set to enabled and this interface can be configured with multiple static IPs along with a DHCP-configured IP. However this has to be done with the netsh add address command, not yet possible via the GUI.

Little documentation has been published about the update to netsh and there is little discussion about it on the entire Internet (Google search dhcpstaticipcoexistence gives only 5 results at the time of this answer). But this feature is real.

Solution 2

Here are the exact commands required for Cecil's suggestion based on Windows 10's new dhcpstaticipcoexistence feature:

1. Find out the interface name:

   netsh interface ipv4 show interface
   

2. Enable dhcpstaticipcoexistence:

   netsh interface ipv4 set interface interface="interface name" dhcpstaticipcoexistence=enabled
   

3. Add a static ip address to your interface

   netsh interface ipv4 add address "interface name" 192.168.x.xxx 255.255.255.0
   

4. Use command 'ipconfig /all' to verify the static ip address is added. You can not do it from GUI.

Solution 3

I wrote a small batch-file. You can test to see if it works in your situation. (here it works fine)

• It will set your interface back to DHCP.
• After that it will extract IP, Subnet Mask, Default Gateway and the first DNS-server. This is the difficult bit. Especially if you have multiple interfaces. If it doesn't work we need to fiddle here a bit to get it to work.
• If you want the second DNS too it should be added here (but i didn't look that far because one DNS should be fine for now).
• It will set these settings "static" on the interface.
• After that you can add the 10.x.y.z address to your interface without a problem.

Here is the script:

@echo off
set interface="Ethernet 2"
set extra_ip=10.0.0.33
set extra_mask=255.255.248.0

echo Setting %interface% back to DHCP
netsh int ipv4 set address name=%interface% source=dhcp
netsh int ipv4 set dnsservers name=%interface% source=dhcp

echo Waiting for IP to stabilize...
timeout /t 5

echo Getting current IP of %interface%
ipconfig > %temp%\ipconfig.txt
for /f "tokens=2 delims=:" %%a in ('type %temp%\ipconfig.txt ^| find "IPv4" ^| find /v "127.0"') do set _IP=%%a
set IP=%_IP:~1%
for /f "tokens=2 delims=:" %%a in ('type %temp%\ipconfig.txt ^| find "Subnet" ^| find /v "127.0"') do set _IP=%%a
set MASK=%_IP:~1%
for /f "tokens=2 delims=:" %%a in ('type %temp%\ipconfig.txt ^| find "Default" ^| find /v "127.0" ^| find /v "::" ') do set _IP=%%a
set GATE=%_IP:~1%
for /f "tokens=2 delims=:" %%a in ('type %temp%\ipconfig.txt ^| find "DNS Servers" ^| find /v "127.0"') do set _IP=%%a
set DNS1=%_IP:~1%

echo Setting IP Address, Subnet Mask and Default Gateway...
echo (IP %IP%, mask %MASK%, gw %GATE%)
netsh int ipv4 set address name=%interface% static %IP% %MASK% %GATE% gwmetric=1

timeout /t 5
echo Setting Primary DNS (%DNS1%)...
netsh int ipv4 set dnsserver name=%interface% static %DNS1% primary

echo Adding secondary IP...
netsh int ipv4 add address %interface% %extra_ip% %extra_mask%

timeout /t 5
echo.
echo New IP configuration:
ipconfig /all

del %temp\ipconfig.txt

You only need to run this once a period of your lease (or after a restart). So if your lease is 10 days you could set this in the task scheduler for 3 AM on Sunday and after every restart. If your computer is always off at night it would only be needed after restart.

Solution 4

One workaround could be to run a VM on the same machine and add a static IP to that VM. This will essentially provide a virtual adapter on the host but for the 10.x.v.z network.

Custom Networking configufation simular to the one you are requesting is well docunmented here.

Solution 5

How about scheduling a script to add the IP address after boot:

netsh interface ipv4 add address “Local Area Connection” 10.x.y.z 255.255.255.0

View more solutions

Comments

[Gustavo Sarmiento]

Great idea, but I am confident that force the adapter out of DHCP.

[Vedant]

The VM idea is hunting butterflies with a sledgehammer. Adding a VM system and licensing Windows for VM use would be complicated and probably more expensive than just adding a second NIC to the machine. I only need the 10.x.y.z address to serve a single program.

[Vedant]

I tested it, and @Fergus is right: saying netsh interface ipv4 show address after this gives the static IP only. The DHCP IP is gone.

[Paul]

Ooops, yeah I remember now. The option would be to get the current IP, then add both back. It is very unlikely that an DHCP lease would not be renewed while in actively in use (there is a grace period to avoid this), so it only matters at boot time.

[Vedant]

@Paul: If I just add the DHCP address back with netsh, doesn't that make it a static IP, so the Windows box's stack won't keep sending DHCP lease renewals to the DHCP server? When the initial lease time is up, the DHCP server would then be free to reassign that address.

[Gustavo Sarmiento]

That is why I preface the answer with "one workaround could be.."; adding a VM would certainly work. Multihoming is obvious; but then again, your idea would fundamentally change the original question. good luck

[Rik]

@WarrenYoung How long is the lease you are getting? (ipconfig /all. for me it's 10 days from my router). So if the computer is restarted before the lease is over it shouldn't be a problem. Otherwise you could add a ipconfig /release and ipconfig /renew above these lines and schedule it to run at 3 AM (when nobody is behind the computer).

[Rik]

@WarrenYoung Oops. When you did a netsh to add an ip you can't use ipconfig /renew anymore because the interface doesn't do dhcp anymore. You need to do netsh interface ip set address name="Local Area connection" source=dhcp. This will set it back to dhcp and get a new lease at the same time. After which you can do the add of 10.x.y.z and then again for your received ip from the lease. You can wrap this al in a renew.bat which you run at 3 AM.

[Paul]

@Rik Yeah, that is a way, Warren needs to check the lease times to ensure the renewal period isn't greater than the lease, but that is just adjusting the frequency.

[paquetp]

This looks...so...shaddy....

[jljtgr]

This answer was very useful to me as it worked across reboots and seemed like a good 1st party solution. Including examples of the commands to use would have been nice. This answer was so close to giving example commands, but I had to go to Google to figure out the exact syntax. My final commands were netsh interface ipv4 set interface interface="Ethernet" dhcpstaticipcoexistence=enabled and netsh interface ipv4 add address "Ethernet" 192.168.1.200 255.255.255.0.

[Dave Ruske]

Very helpful! Is there an API somewhere to enable dhcpstaticipcoexistence, preferably something not too hideous to call from C++? Haven't located anything yet in the IP Helper documentation. I prefer not to shell executables as a substitute for an API.

[Admin]

Thanks I've been working round this issue for years. Glad to finally find a solution.