Connection issue with the August Lock Pro gen3

[aeozyalcin]

Hello,

I love what you did with the python interface, unfortunately I couldn't get it to work on my end. I got the offline key and index with a rooted android phone, however when I issue the command "sudo python3 cli.py front --unlock", I get the following error messages:

Writing command: xxxxxxxxxxxx
Encrypted command: yyyyyyyyyyyyy
Traceback (most recent call last):
File "cli.py", line 38, in
lock.connect()
File "/home/pi/augustpy/augustpy/lock.py", line 55, in connect
response = self.secure_session.execute(cmd)
File "/home/pi/augustpy/augustpy/session.py", line 96, in execute
return self._write(command)
File "/home/pi/augustpy/augustpy/session.py", line 89, in _write
self.peripheral.waitForNotifications(10) is False:
File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 560, in waitForNotifications
resp = self._getResp(['ntfy','ind'], timeout)
File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 407, in _getResp
resp = self._waitResp(wantType + ['ntfy', 'ind'], timeout)
File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 362, in _waitResp
raise BTLEDisconnectError("Device disconnected", resp)
bluepy.btle.BTLEDisconnectError: Device disconnected

Any clue what might be going on? Thanks!

[aeozyalcin]

I actually have it boiled down to "self.peripheral.waitForNotifications(10)". Bluepy seems to think that the lock is maybe disconnecting while it's waiting for a reply. This is happening during the first characteristic write Send SEC_LOCK_TO_MOBILE_KEY_EXCHANGE. I don't know enough about the interface to tell if it's an issue with encryption, or if it's related to the newer locks?

Any idea what might be the issue? I know I have the right offline key/index (obtained via rooted Android phone). I am really hoping you can help!

[aeozyalcin]

Some more updates, I dug deeper with debugging the BLE connection using the nRF connect app. When I connect to the lock with the app, and attempt to write something to the secure write characteristic of the lock, the lock will initially disconnect from BLE. This is what I think I am seeing exactly happening when using the augustpy library as well., which is a bummer. I am wondering if this a Lock Pro 3rd gen specific behavior. I would really love to see this work. Any suggestions are welcome.

[aeozyalcin]

Another update, the issue ended up being due to the offline key I was using. It didn't like my offline key at index 0 for some reason. When I switched to using girlfriend's key, I actually got it to work. I am really excited about the potential that this python library offers. However, one snag I am hitting is that when the connection is interrupted unexpectedly for some reason, next time I try to connect to the lock via python fails. I noticed that I have to then connect using the app on the phone once, which seems to "reset" something in the lock, and then I can connect again using the python library. Now onto figuring that bit out :/

[aeozyalcin]

I got it!!! The connection is rock solid now, I can connect and operate the lock any way I lock each time I want. I noticed that @Friendly0Fire mentioned about unreliable connection in the last commit, I think I figured out the issue.

So the issue was that I was randomly able to connect just fine and operate the lock. But then the next day when I try to connect, I would write the handshake key to the lock but receive no response back from the lock. I figured out that if I connect once using my phone with the app, then I was able to get the RPi to connect to the lock just fine without problems for a while, but then eventually my RPi would stop being able to connect.

It turns out, that during the connection process, we are not subscribing for indications from the SEC or the MCU. So what was happenning was, we write the handshake key, the lock updates the read register with its response, however BluePY was not getting the message because we hadn't subscribed for the indications. And hence why connection was failing. Furthermore, the reason why connecting with my phone made my RPi work for a while was that the phone was enabling all the indications on all the UUIDs on the lock, so when I connected with my RPi later, I was getting the indications. Until the lock turned off all notifications/indications. After that point when I try to connect and write the handshake key, I wasn't getting anything back.

Long story short, here is how you fix it:

                response = self.peripheral.writeCharacteristic(26, b'\x02\x00', withResponse=True)
                print("Subscription SEC request response: ",response)

                response = self.peripheral.writeCharacteristic(21, b'\x02\x00', withResponse=True)
                print("Subscription MCU request response: ",response)

These lines needs to go into lock.py inside the connect function to line 47, right after we retreive device characteristics, and before we set the sessions key. First line enables indications on the SEC, and the second line enables indications on the MCU.

I am so glad I got this figured out... I have spent all week on this and now this library will allow me to automate my home just like I wanted it. 😃

[Friendly0Fire]

First, thank you for pinging me directly. For whatever reason, Github hadn't auto-watched this repo for me, so I wasn't getting notifications. Sorry about that!

Second, that's amazing news, I'm glad you figured it out! I'll merge this change ASAP. I'd sort of given up on this project with how unreliable and inconsistent the connection was, so hopefully that'll be the change it needed!

[aeozyalcin]

No problem! Thank you for putting together the library. Please do let me know if this fixes your unreliable connection issue.

I have a 3rd gen lock, not sure which gen you have. The characteristic handles I am using are hard coded (26 and 21) and may not be the same for you. I initially tried to derive the handles dynamically by searching for descriptors under the characteristics, but BluePy was taking too long, and the lock was timing out and disconnecting (5 seconds of inactivity causes the lock to terminate connection). So to speed it up, the handles are currently hard coded. Let me know how it goes!

[aeozyalcin]

@Friendly0Fire tagging you in case you didn't see my last message. Thanks.

[benaranguren]

@aeozyalcin how did you find the char handles?

[aeozyalcin]

@aeozyalcin how did you find the char handles?

            self.command_service = self.peripheral.getServiceByUUID(self.COMMAND_SERVICE_UUID) 
            characteristics = self.command_service.getCharacteristics()
            #descs = characteristic.getDescriptors()
            for characteristic in characteristics:
                if characteristic.uuid == self.WRITE_CHARACTERISTIC:
                    self.session.set_write(characteristic)
                elif characteristic.uuid == self.READ_CHARACTERISTIC:
                    self.session.set_read(characteristic)
                    #descs = characteristic.getDescriptors()
                    #for desc in descs:
                        #print("found  desc: " + str(desc.uuid))
                        #str_uuid = str(desc.uuid).lower()
                        #if str_uuid.startswith("00002902"):
                            #mcu_sub_handle = desc.handle
                            #mcu_sub_handle = 21
                            #print("*** Found MCU subscribe handle: " + str(mcu_sub_handle))
                elif characteristic.uuid == self.SECURE_WRITE_CHARACTERISTIC:
                    self.secure_session.set_write(characteristic)
                    print("Set Secure Write")
                elif characteristic.uuid == self.SECURE_READ_CHARACTERISTIC:
                    self.secure_session.set_read(characteristic)
                    print("Set Secure Read")
                    #descs = characteristic.getDescriptors()
                    #for desc in descs:
                        #print("found  desc: " + str(desc.uuid))
                        #str_uuid = str(desc.uuid).lower()
                        #if str_uuid.startswith("00002902"):
                            #sec_sub_handle = desc.handle
                            #sec_sub_handle = 26
                            #print("*** Found SEC subscribe handle: " + str(sub_handle))

Here is the code I wrote to find the handles, it goes into the lock.connect() function. Once I determined it was taking too long for my liking, I commented out most of it, and used hard coded in the notification subscription requests. Hardcoded values work to this date. You will need to uncomment most lines, clean it up a bit and it should print out the handles for you. Let me know how it goes, good luck!

[benaranguren]

Thanks @aeozyalcin. It's been working solid for me too with your patch. I'm not familiar with BLE, so how did you know to look for 00002902?

[benaranguren]

Oh wait ... I see in nRF app that 0x2902 is listed as Client Characteristic Configuration UUID: 0x2902. That must be the link.

[aeozyalcin]

@benaranguren yep you got it. 00002902 has some sort of significance in BLE, the readable/indicatable characteristics typically have that descriptor.

That's awesome, I am glad to hear I've contributed! This project taught me a lot about BLE, very fun and rewarding. I have made many more modifications to FireFly's library, and I hope to share it all one day once I finish the "grand" project that I have been working on.

[benaranguren]

One more question @aeozyalcin . Where did you get b'\x02\x00' ?

[aeozyalcin]

@benaranguren if you look at nRF Connect debug output as you have the August app running, that's what we are writing initially to the characteristic to subscribe to indications. I think 02 is indications, and 01 is notifications (might be the other way around).

[benaranguren]

Pretty cool. Thanks.

[Flole998]

@aeozyalcin Did you figure out how the lock "pushes" state changes? The connect is able to send push messages to the phone, so somehow the lock has to publish changes. I tried your patch and added a waitForNotifications aswell as a Handler but that didn't help to get notifications when the state changes unfortunately, do you have any additional information?

[aeozyalcin]

@aeozyalcin Did you figure out how the lock "pushes" state changes? The connect is able to send push messages to the phone, so somehow the lock has to publish changes. I tried your patch and added a waitForNotifications aswell as a Handler but that didn't help to get notifications when the state changes unfortunately, do you have any additional information?

I have that working on my end, but it required a fundamental change to how this library is implemented. You need to go into multithreading, and have a thread that is always waiting for notifications, and feeds a queue that another thread consumes and reacts to. I had to significantly grow the classes to add that functionality.

[Flole998]

Could you email me your copy, or even better publish it on GitHub? I don't really understand why my lock is not pushing the changes and dropping the connection after some time....

[Flole998]

@aeozyalcin Any chance to get a copy of your implementation? I still can't figure out why exactly my lock is dropping the connection and not sending notifications for status changes, I've tried a multithreaded approach now aswell (I just want to publish to MQTT, so no need for a queue there, the listener thread can directly publish it) but I think I'm doing something wrong during the connection initiation that causes the lock to disconnect.

[Flole998]

@aeozyalcin I am able to get notifications from the lock now when it's locked/unlocked, however I am still having issues with periodic disconnects. After 90 seconds the lock closes the connection. What do I need to do to prevent that? Or do I just reconnect in that case (but I think that would increase battery usage)?

[aeozyalcin]

@aeozyalcin I am able to get notifications from the lock now when it's locked/unlocked, however I am still having issues with periodic disconnects. After 90 seconds the lock closes the connection. What do I need to do to prevent that? Or do I just reconnect in that case (but I think that would increase battery usage)?

You need to keep the connection up by sending dummy request. I ask for lock status every 60 seconds to keep the lock up. With the latest lock FW, 90 is indeed the default timeout setting I have seen. Depending on which library you are using to control your BLE stack, you can lower your connection interval and increase latency so your lock stays low power for longer. I am directly sending HCI messages to Bluez, as I found BluePy to not give as much low level control as I like.

[Flole998]

Alright, then my implementation is probably very similar to yours. I had to raise the interval aswell because otherwise the lock seemed to stop responding after a while. Now I am still seeing a connection drop every 24 hours, I am not sure what exactly causes that. I assume you don't have that issue?

I will switch to pygatt which allows setting minimum/maximum interval aswell as latency and generally seems to be more robust. What values are you using for interval/latency?

I reverse engineered the native library from the App aswell to extract all the possible status codes from the lock (and also to see what commands are possible), and I am frequently seeing codes 0x00 and 0x01 (which are init and calibrating) for some reason. I'm ignoring those for now. What is really weird is, that the lock isn't allowing a simple unlock, it always unlatches aswell. So if I just want to unlock but keep the door shut then it doesn't work.

[tongaimaramba]

Hi @Flole998 ! You seem to have got your implementation going pretty well. I was wondering if you had any tips - my setup keeps falling over when trying to get characteristics:

 lock.connect()

File "/home/pi/py_apps/august/augustpy/lock.py", line 36, in connect
characteristics = self.command_service.getCharacteristics()
File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 144, in getCharacteristics
self.chars = [] if self.hndEnd <= self.hndStart else self.peripheral.getCharacteristics(self.hndStart, self.hndEnd)
File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 508, in getCharacteristics
rsp = self._getResp('find')
File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 407, in _getResp
resp = self._waitResp(wantType + ['ntfy', 'ind'], timeout)
File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 362, in _waitResp
raise BTLEDisconnectError("Device disconnected", resp)
bluepy.btle.BTLEDisconnectError: Device disconnected

I also wasn't sure whether the patch that @aeozyalcin came up with is applied inside or outside the For loop that retrieves the device characteristics. How did you implement this?

Many thanks!

[Flole998]

Use hcidump to see what's going on

[tongaimaramba]

Use hcidump to see what's going on

I'm very new to BLE but it seems like the lock is refusing connection. Appreciate any suggestions for troubleshooting. FYI I've triple checked the mac address, restarted adapters, reconnected via the app and tried changing the keyindex from 1 to 0 but no success.

Here's an extract from hcidump:

HCI Event: Command Complete (0x0e) plen 4
LE Set Scan Enable (0x08|0x000c) ncmd 1
status 0x00
HCI Event: Command Complete (0x0e) plen 4
LE Add Device To White List (0x08|0x0011) ncmd 1
status 0x00
HCI Event: Command Complete (0x0e) plen 4
LE Set Scan Parameters (0x08|0x000b) ncmd 1
status 0x00
HCI Event: Command Complete (0x0e) plen 4
LE Set Scan Enable (0x08|0x000c) ncmd 1
status 0x00
HCI Event: Command Complete (0x0e) plen 4
LE Set Scan Enable (0x08|0x000c) ncmd 1
status 0x00
> HCI Event: Command Status (0x0f) plen 4
LE Create Connection (0x08|0x000d) status 0x00 ncmd 1
HCI Event: Command Status (0x0f) plen 4
LE Read Remote Used Features (0x08|0x0016) status 0x00 ncmd 1
HCI Event: Command Complete (0x0e) plen 4
LE Set Scan Enable (0x08|0x000c) ncmd 1
status 0x0c
Error: Command Disallowed
HCI Event: Command Complete (0x0e) plen 4
Reset (0x03|0x0003) ncmd 1
status 0x00

Also:
I tried to connect via gatttool and failed (but not sure if this because of security level?):

[LE]> connect
Attempting to connect to NN:NN:NN:NN:NN:NN
Connection successful
[LE]>
(gatttool:9354): GLib-WARNING **: 17:32:50.524: Invalid file descriptor.
[LE]> primary
Command Failed: Disconnected
[LE]>

[Flole998]

No, the lock isn't doing anything. Most likely your adapter is broken (broken as in broken in Linux, so a driver issue). There are a few ones out there which have broken support.

[tongaimaramba]

No, the lock isn't doing anything. Most likely your adapter is broken (broken as in broken in Linux, so a driver issue). There are a few ones out there which have broken support.

Yup, you're right! I got a new bluetooth dongle and its now getting further into the script. I'm now stuck at exactly the same place as @aeozyalcin once was "during the first characteristic write Send SEC_LOCK_TO_MOBILE_KEY_EXCHANGE".

I was wondering if the issue was that we shouldn't wait for a response (if the lock disconnection is expected) so I tried "self.write_characteristic.write(command, False)" in the _write function of sessions.py. But it made no difference. I even tried keyIndex 2 instead of 1, but no joy.

So it seems like the lock disconnects when I submit the command to do the first secure write. I see the same behaviour if I try the command in interactive mode in Gatttool.

Below is the trace output (I had debug enabled for bluepy.

Appreciate any ideas!

Trace output

Sent: wr 17 ab216013de405275ea512ce844d38c730f01

Got: 'rsp=$wr\n'
Got: "rsp=$stat\x1estate=$disc\x1emtu=h0\x1esec='low\n"
Stopping /usr/local/lib/python3.7/dist-packages/bluepy/bluepy-helper
Traceback (most recent call last):
File "cli.py", line 39, in
lock.connect()
File "/home/pi/py_apps/august/augustpy/lock.py", line 62, in connect
response = self.secure_session.execute(cmd)
File "/home/pi/py_apps/august/augustpy/session.py", line 99, in execute
return self._write(command)
File "/home/pi/py_apps/august/augustpy/session.py", line 91, in _write
self.peripheral.waitForNotifications(10) is False:
File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 586, in waitForNotifications
resp = self._getResp(['ntfy','ind'], timeout)
File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 427, in _getResp
resp = self._waitResp(wantType + ['ntfy', 'ind'], timeout)
File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 382, in _waitResp
raise BTLEDisconnectError("Device disconnected", resp)
bluepy.btle.BTLEDisconnectError: Device disconnected

Here is the hcdump

HCI Event: Command Complete (0x0e) plen 4
LE Set Scan Parameters (0x08|0x000b) ncmd 1
status 0x00
HCI Event: Command Complete (0x0e) plen 4
LE Set Scan Enable (0x08|0x000c) ncmd 1
status 0x00
HCI Event: Command Complete (0x0e) plen 4
LE Set Scan Enable (0x08|0x000c) ncmd 1
status 0x00
HCI Event: Command Status (0x0f) plen 4
LE Create Connection (0x08|0x000d) status 0x00 ncmd 1
HCI Event: Command Status (0x0f) plen 4
LE Read Remote Used Features (0x08|0x0016) status 0x00 ncmd 1
HCI Event: Disconn Complete (0x05) plen 4
status 0x00 handle 75 reason 0x13
Reason: Remote User Terminated Connection

[aeozyalcin]

How did you retrieve your key? Original issue I had was that I somehow had the wrong key. @tongaimaramba

[tongaimaramba]

Well now I feel foolish! Thanks for making me check @aeozyalcin! I had taken the key from a rooted android phone. I went back to check again and found a completely different key and index. I'm sure I copied them correctly before, I guess there must be scenarios where the key changes. Have you come across this before?

Anyway, thanks to you, and to @Flole998 for the help. Looking forward to playing around with this!

[aeozyalcin]

@tongaimaramba no worries, I'm glad you got it working. The key will change if you log in using a different phone. One thing I recommend is to create an August account specifically for your device that's going to be controlling the lock using this library. Once you acquire the key and the index, log out of that account, and just never log in with it again on your phone. The key will remain in the lock pretty much forever and you won't have to deal with changing keys.

[tongaimaramba]

@Flole998 would you be able to share the command codes you managed to find? I'm trying to replicate the "open" function that is available in the app. i.e. lock is unlocked state but latch is on - I want get it to unlatch so the door can be opened. Currently, if I call 'unlock' (0x0a) while the lock is unlock state, but latched, it does nothing. Is it possible I have an error in my config?

[tongaimaramba]

@Flole998 would you be able to share the command codes you managed to find? I'm trying to replicate the "open" function that is available in the app. i.e. lock is unlocked state but latch is on - I want get it to unlatch so the door can be opened. Currently, if I call 'unlock' (0x0a) while the lock is unlock state, but latched, it does nothing. Is it possible I have an error in my config?

Update: I found it. I can use lock.force_unlock() to do this. Thanks!

[Flole998]

With the latest lock FW, 90 is indeed the default timeout setting I have seen.

I got a new USB-Bluetooth dongle today as I continued to experience issues with the old one and also my lock firmware was updated recently. Is there a new timeout? My lock drops the connection now after 30 seconds, are you seeing the same behaviour? Which firmware is your lock running?

[CrazeeGhost]

For anyone that may stumble upon here looking for the connection issue fix. I've created a fork and implemented the fix suggested by @aeozyalcin in one of the comments above. See https://github.com/CrazeeGhost/augustpy

[aeozyalcin]

Hey all, I finally got around to publishing the code in the form of an August BLE to MQTT Bridge.
aeozyalcin/August2MQTT

[Flole998]

After more than 2 years I have my own version which is more complete than your version. How did you figure out how the voltage report data is formatted? It doesn't match what I've reverse engineered from the android App.

[aeozyalcin]

Look here
https://github.com/aeozyalcin/August2MQTT/blob/679cc0c348c43ec8deb9ea7b4f34f809bf5713b4/augustpy/lock.py#L404

[Flole998]

That does not match what the official App is doing in my opinion. So how did you figure that out? Just trial and error?