User Tools

Site Tools


Navigation Menu

Flight-Control

  • StatusClosed
  • OP-ModePre-Launch
  • LocationN48 - E11
  • Localtime20:25
  • CountdownT-00D 00:00

Hot Projects

SEEDStack

SEEDStack - Open 3D printable seed/sprouting systemDIY Food Hacking

UCSSPM

UCSSPM - Unified Clear-Sky Solar Prediction ModelOpen Solar Power

picoReflow

picoReflow - DIY PID Reflow Oven Controller based on RaspberryPiDIY Reflow Soldering

PiGI

PiGI - DIY Geiger Counter based on RaspberryPiRasPi Geiger Counter

DIY ARA-2000

Active Wideband Receiver Antenna for SDR - ARA-2000Wideband Antenna

DSpace

DSPace - Map everythingMap everything!

Mission-Tags

Harder Soft-Unbricking a Ubiquiti Unifi UAP-Pro AP

ubiquiti-unifi-uap-pro-inside-with-serial-debug-connected

Re-deploying the Ubiquiti Unifi UAP-Pro with OpenWRT (instead of Unifi) at the new base-camp revealed a problem: it simply didn't work anymore. No DHCP reaction, no default IP, nothing. Except for the LED blinking white after a reset, it seemed completely broken. Opening the case and attaching a terminal to the serial console, revealed bootloader output that wasn't promising but at least it wasn't completely dead. So we had to get our hands dirty, pop the hood and hack around to fix this jffs2 corruption issue and revive the AP without costly and probably long/painful RMA. Maybe this knowledge can help you too.

The Symptom

U-Boot unifi-v1.2.3.74-ge597862a (Mar 14 2013 - 18:30:11)

Board: Ubiquiti Networks AR9344 board (e507-27.2122.0030.0030)
DRAM:  128 MB
Flash: 16 MB
Net:   eth0
Hit any key to stop autoboot:  0
Scanning JFFS2 FS: .| 
Unknown node type: e008 len 52 offset 0xf41eec
[...]                                                                                                                                       done.
cant find .firewall.uci-mWgTyx��
[...]
cant find .wireless.uci-HxpWh9
cant find .network.uci-anmaB1
�����������������������������������������������������������������������
�����������������������������������������������������������������������
�����������������������������������������������������������������������
�������������������������������������������������������

It is still unclear what exactly causes this jffs2 corruption, but there have been at least three other reported cases and one directly confirmed by djs500, who found the draft-wip-template for this mission log on the net, had exactly the same problem and came back with positive feedback that he was able to fix it using Solution 3, as we did too.

Solution 1

https://community.ubnt.com/t5/UniFi-Troubleshooting/UniFi-TFTP-soft-recovery-for-bricked-access-point/ta-p/607605

Set a static IP on your PC's NIC from 192.168.1.0/24 range, but not 192.168.1.20 (this is the Unifi AP default TFTP IP).

Follow the steps to unbrick your UAP:

  1. Set the IP on your PC to be able to access the bricked unit and prepare the firmware file.
  2. On your PC, open your TFTP client and locate the firmware.bin file so you can start it later. Do not initiate the transfer.

Using the Windows integrated TFTP client the command to prepare is: “tftp -i 192.168.1.20 PUT [path to file]\firmware.bin” Unplug the bricked unit. Plug the LAN connection of the PoE injector directly to your PC's NIC. Keep the UniFi AP's reset button depressed and plug in network/PoE in the unit. Keep the reset button depressed until you see the light cycling relatively fast through amber/green/off colors (~14 seconds from power on) → Release it. Now the device is in TFTP transfer mode. Engage the TFTP push command and wait. The device will write the firmware and it will reboot. If you wait too long to start the TFTP transfer, the push will not work as the device will stall. Please redo from step 3. To ensure all goes clean, after the device gets stable (blinking or steady amber LED), give it a reset: Remove power. Reconnect holding the reset button for ~7 seconds (green led will flash once) → release reset button and wait for the device to stabilize again. After device is stabilized, power cycle it again, and you are good to go on adopting the device in the controller again.

Official UBNT solution didn't work at all

This may be due to the fact that this workflow expects a bricked UBNT firmware and NOT an OpenWRT installation.

Solution 2

Soft Unbrick via Serial using urescue and a tftp server was another method mentioned somewhere, but sadly it never really flashed anything and went staight to the reset in less than a second instead. It could never have written the complete firmware in this short time period and after testing: it didn't, so “Firmware update complete” was a lie:

ar7240> urescue
Setting default IP 192.168.1.20
Starting TFTP server...
Using eth0 (192.168.1.20), address: 0x81000000
Waiting for connection: -
Receiving file from 192.168.1.254:39067
Received 4683453 bytes
Firmware Version: BZ.ar934x.v3.2.1.2601.140606.1622
Setting U-Boot environment variables
Will not overwrite u-boot partition! Skipped.

Firmware update complete.

Resetting..

Serial urescue didn't work at all

Solution 3

None of the above solutions worked with our OpenWRT UAP-Pro, so we had to grab one of the original jffs2 images from a live one with original UBNT firmware, use a tftp server to transfer that jffs2 image file to the AP, unlock and erase the corrupt jffs2 image on the flash chip and eventually write the new image by hand.

The following unbrick trace documents these 9 easy to follow steps codec, fpletz and I took to restore the UAP to its orginal UBNT firmware with default configuration (and default ubnt login) to be able to finally reinstall OpenWRT.

Step 1: Preparation

If you've lost your original firmware backup too, you can download the jffs2 image here:
jffs2.img.bz2

Just bunzip2 the downloaded jffs2.img.bz2 and put jffs2.img into your tftproot so that the AP can access it via LAN. If you don't know which tftp-server to pick, dnsmasq and atftp have been reliable alternatives and have a solid base of documentation and HOWTOs to get started out there.

In this setup dnsmasq with tftp enabled was hosted on a laptop, eth0 IP 192.168.1.254 connected via LAN to the main port of the UAP-Pro.

Step 2: Connect Serial Console

unifi-uap-pro-inside-ttl-serial-connection-info

Step 3: Transfer jffs2 image to AP

Power up the AP and press a key to enter u-boot menu, then transfer and load the original jffs2 image over tftp to the AP and store it in RAM, starting at memory address 0x83000000:

ar7240> tftp 83000000 jffs2.img

If the tftp server is reachable, the output should look like this:

Using eth0 device
TFTP from server 192.168.1.254; our IP address is 192.168.1.20
Filename 'jffs2.img'.
Load address: 0x83000000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #############################
done
Bytes transferred = 16121856 (f60000 hex)

The firmware image is stored in RAM now and occupies 0xf60000 bytes. Writing it from there to flash memory directly may fail when the target area has not been erased or is write-protected. Let's take care of that first.

Step 4: Reset partition table to defaults

This gives you an overview of the partition table layout

ar7240> mtdparts
device nor0 <ath-nor0>, # parts = 5
 #: name                        size            offset          mask_flags
 0: u-boot                      0x00040000      0x00000000      0
 1: u-boot-env                  0x00010000      0x00040000      0
 2: jffs2                       0x00f60000      0x00050000      0
 3: cfg                         0x00040000      0x00fb0000      0
 4: EEPROM                      0x00010000      0x00ff0000      0

active partition: nor0,0 - (u-boot) 0x00040000 @ 0x00000000

defaults:
mtdids  : nor0=ath-nor0
mtdparts: mtdparts=ath-nor0:256k(u-boot),64k(u-boot-env),15744k(jffs2),256k(cfg),64k(EEPROM)

Reset partition table to defaults with

ar7240> mtdparts default

Save environment changes:

ar7240> saveenv
Saving Environment to Flash...
Un-Protected 1 sectors
Erasing Flash.... done
Erased 1 sectors
Writing to Flash... write addr: 9f040000
done
Protected 1 sectors

Step 5: Unlock Flash-Banks

Now it is time to make sure that any write-protection is disabled:

ar7240> protect off all
Un-Protect Flash Bank # 1

Step 6: Erase the corrupt jffs2

Figuring out the correct address and length was a bit difficult, unclear, uncertain, not well documented and took us quite a while, because we didn't want to brick it more by writing stuff somewhere other than we actually intended. Unfortunately, there was no documentation during the process of how we derived at that conclusion, but in the end we finally decided to start at address 0x9f050000 for the length of the jffs2 image as indicated by the tftp transfer (0xf60000) and the mtd partition size (0xf60000).

ar7240> erase 0x9f050000 +0xf60000
...................................................................................................................................................................................................................................................... done
Erased 246 sectors

Step 7: Flash original jffs2

In this step we're going to copy the jffs2 image stored in RAM, starting at address 0x83000000, and write it to the previously erased jffs2 flash area starting at address 0x9f050000 for the length of 0xf60000. The cp command knows about flash memory areas and will automatically invoke the necessary flash programming algorithm, when the target area is in flash memory.

ar7240> cp.b 0x83000000 0x9f050000 0xf60000
Copy to Flash... write addr: 9f050000
done

Step 8: Verify the jffs2 was flashed correctly

Since we still have the jffs2 image stored in RAM, starting at address 0x83000000, we can now comfortably byte-compare it with the content starting at address 0x9f050000 (Flash) for the length of 0xf60000. Both have to match, otherwise something obviously went wrong during the flashing process.

ar7240> cmp.b 0x83000000 0x9f050000 0xf60000
Total of 16121856 bytes were the same

Step 9: Reset, Reinstall OpenWRT and enjoy your revival work

ar7240> reset
Resetting...

U-Boot unifi-v1.2.3.74-ge597862a (Mar 14 2013 - 18:30:11)

Board: Ubiquiti Networks AR9344 board (e507-27.2122.0030.0030)
DRAM:  128 MB
Flash: 16 MB
Net:   eth0
Hit any key to stop autoboot:  0
Scanning JFFS2 FS: . done.
## Booting image at 81000000 ...
   Image Name:   MIPS Ubiquiti Linux-2.6.32.33
   Created:      2014-06-06  23:26:31 UTC
   Image Type:   MIPS Linux Kernel Image (lzma compressed)
   Data Size:    4450109 Bytes =  4.2 MB
   Load Address: 80002000
   Entry Point:  80002000
   Verifying Checksum at 0x81000040 ...OK
   Uncompressing Kernel Image ... OK

Starting kernel ...
Mission-Log entry created by chrono on 2016/03/18 15:56 UTC

Discussion

Tolsi
2016/04/13 09:49

Thanks for this post, it is very useful!

Please, can you post your 'printenv' output?

chrono
2016/04/16 07:39

I'm sorry, but our UAP has already been re-flashed with OpenWRT. re-assembled and installed under the ceiling in active duty. But if there ever is a reason to get it down again, I'll make sure to fetch that. Why do you need it?

In the meantime, maybe someone else is hanging out there with a serial connector on the UAP's PCB right now, who could supply the env?

Tolsi
2016/04/21 20:05

My access point is constantly loaded in the flash mode, even without holding the button, and I changed the some environment vars, now I sometimes see the message that it has been damaged, but it does not interfere and it helps.

I have another problem. We tried OpenVRT on this device and we were not satisfied with his work and I want to return to the standard firmware. I flashed the firmware from the official site (BZ.ar934x.v3.3.20.4019.160401.1453.bin), but now after loading it quickly blinks white LED, it gets DHCP address and have open SSH port, but the user and password ubnt/ubnt not suitable. Maybe I need your password (the latest username and password, which was set by the controller), from the AP from which loads dumped firmware? Can you send it to my email or smtn? I can not connect AC to the controller and I can't do nothing with it.

chrono
2016/04/23 07:25

Have you tried https://help.ubnt.com/hc/en-us/articles/205143490-UniFi-How-to-reset-the-UniFi-Access-Point-to-factory-defaults to reset the UAP after going through the above tutorial? Since all parts on the MTD should at this point be pure Unifi, it should reset and be re-adoptable by the UniFi Controller and accessible via ubnt/ubnt.

Hard to debug from here and Ubiquiti has gone out of there way to lock down the bootloader to make open firmware (like OpenWRT) installation extremely hard/impossible. When a manufacturer is not willing to support an open community, it's hard to see why the open community should offer support for locked down hardware.

Johan
2016/06/09 16:33

Thanks this saved my day.

chrono
2016/06/09 20:42

Great, thanks for the feedback. It's always good to know it worked out for more people, to justify the time that was invested in solving, documenting and sharing this issue. So that others may find and use it, to save their own valuable lifetime, increase p2p knowledge-transfer speed/efficiency/proficiency and collectively reduce frustration levels for all of us :)

Kerwindena
2016/06/27 19:08

Hey,

can you tell me, how to open the device? I have read about 4 torx screws on the bottom of the device, but mine does not hav any screws although there are 4 indentations in the positions of the screws. Anybody knows if it is glued or cliped together and has any suggestions on how to open it?

Kerwindena
2016/06/27 19:42

The device is actually both. But thankfully the glue is quite thin and seems to be thought of as a seal more than anything else. To open it you have to get your plastic opening tool of your fingernail in the slid around the device and rub the glue out of there for quite a long time. When you finally get it in for about a millimeter go all around with it until the glue seal is completly broken. The devise is also held with 5 clips wich can be easily removed therafter.

chrono
2016/06/27 20:05

Too fast :) But thanks for coming back and reporting.

Maybe ubiquiti has recently not only chosen to lock down their bootloaders, but also to glue the cases as well? That would be another step into the wrong direction. I'd like to believe the seal “as an implied quality/reliability improvement” story but it seems unlikely, since their cases are not designed/protected against any harmful ingress of water at all, so what would be the point?

For reference: All other prior UniFI APs I've had to get dirty with before were easily opened by removing the screws and prying a bit on the side where both parts meet (with fingernails - sometimes not even necessary) and lifting the top-side (where the led ring is) off, leaving the pcb clipped to the bottom/mount-side part. No glue or other opening inhibitors whatsoever…

Mark
2017/03/10 15:36

My Ubiquiti AP-Pro died. Tried your tutorial, but i get a mismatch when copying from ram to winbond flash (step 8). So i guess the chip is bad.

Now i ordered a new one, and have a way of programming. My soldering skills aren't that good, so let's say the chip dies when desoldering; is there a way to start programming a new chip from scratch? (not just the ffsimg, but everything, boot/uboot-env/cfg/eeprom partitions). If yes, how?

Or is there a pre-boot command that can export the entire chip contents?

Comment text

Please type the letters you see on the left into the box on the right:

  _____   ___    _  __   ____  _____
 / ___/  / _ |  | |/_/  / __/ / ___/
/ (_ /  / __ | _>  <   / _/  / (_ / 
\___/  /_/ |_|/_/|_|  /_/    \___/