pickle puzzle

[chan] bitmessage
Feb 11 18:49 [raw]

I tried to unpickle knownnodes.dat, remove all the non-onion entries, then repickle it so that bitmessage will load it properly. I tried with standard pickle commands, edited the json output, but when I repickle it for some reason bitmessage does not recognize it and just erases the file and starts over on startup with the default bootstrap entries. Bitmessage is not able to load the edited pickle. Does any python guru care to lend a hand at what I need to investigate to get the desired result?

[chan] bitmessage
Feb 11 18:59 [raw]

for what?

[chan] bitmessage
Feb 11 19:17 [raw]

the original post already answered this question

[chan] bitmessage
Feb 11 19:29 [raw]

It didn't answer WHY you want to remove the non-onion entries. Especially since BitMessage is just going to get them from the connected nodes again anyway.

[chan] bitmessage
Feb 11 19:36 [raw]

Exactly. Why someone wants to delete non-onion entries?

[chan] bitmessage <<Ext>>
Feb 11 23:34 [raw]

Add to torrc: SocksPort 127.0.0.1:9151 IsolateDestAddr OnionTrafficOnly Then configure bitmessage to use that new port 9151 and it will only connect to onion nodes. Optionally you can delete knownnodes.dat and add to keys.dat: [knownnodes] maxnodes = 1000 This will reduce the number of nodes in knownnodes to 1000, retreive the freshest onion nodes, and connect to those onion nodes them faster. Don't set maxnodes too low or you may never get more than the default boostrap onion node. If you lower maxnodes and don't delete knownnodes.dat, you may end up with 1000 non-onion nodes and never connect at all.

[chan] bitmessage <<Ext>>
Feb 12 00:12 [raw]

import argparse import imp import sys try: import cPickle as pickle except ImportError: import pickle # Dynamically generate and import stripped down 'state' module containing just # the Peer namedtuple to unpickle knownnodes successfully and output a pickle # data stream PyBitmessage understands module_name = 'state' module_code = '''from collections import namedtuple Peer = namedtuple('Peer', ('host', 'port')) ''' module = imp.new_module(module_name) exec module_code in module.__dict__ sys.modules[module_name] = module parser = argparse.ArgumentParser() parser.add_argument('knownnodes', type=argparse.FileType('rb')) args = parser.parse_args() streams = {} for stream, nodes in pickle.load(args.knownnodes).iteritems(): streams[stream] = {peer: attrs for peer, attrs in nodes.iteritems() if peer.host[-6:] == '.onion'} pickle.dump(streams, sys.stdout)

[chan] bitmessage
Feb 12 04:19 [raw]

In much older threads, some have worried that malware could overwrite knownnodes.dat with malicious bootstrap entries. I don't recall the results of the threads, but erasing the file and starting over with the default bootstrap entries would be a good precaution for bitmessage to perform in that situation.

[chan] bitmessage <<Ext>>
Feb 12 05:16 [raw]

If malware is able to modify knownnodes.dat, I'd be more worried about an exploit than malicious bootstrap nodes.

[chan] bitmessage
Feb 12 06:29 [raw]

WHY? none of your business, that's why.

[chan] bitmessage
Feb 12 08:16 [raw]

Add to torrc: SocksPort 127.0.0.1:9151 IsolateDestAddr OnionTrafficOnly Then configure bitmessage to use that new port 9151 and it will only connect to onion nodes. Optionally you can delete keys.dat and add to knownnodes.dat: [knownnodes] maxnodes = 1000 This will reduce the number of nodes in knownnodes to 1000, retreive the freshest onion nodes, and connect to those onion nodes them faster. Don't set maxnodes too low or you may never get more than the default boostrap onion node. If you lower maxnodes and don't delete keys.dat, you may end up with 1000 non-onion nodes and never connect at all.

[chan] bitmessage
Feb 12 11:02 [raw]

I rate it an eight, mate.

[chan] bitmessage
Feb 12 11:03 [raw]

gr8 b8, m8, I r8 8/8

[chan] bitmessage
Feb 12 16:15 [raw]

You think you have superior rationale. The subject is not malware. The subject is how to load the pickle with onion only nodes. It doesn't matter why.

[chan] bitmessage
Feb 12 16:15 [raw]

If I were malware I would send me your keys.

[chan] bitmessage
Feb 12 16:15 [raw]

this is a clever workaround

[chan] bitmessage
Feb 12 16:18 [raw]

> exec module_code in module.__dict__ dangerous vector for injection

[chan] bitmessage <<Ext>>
Feb 12 21:23 [raw]

Not really. An attacker has to be able to modify memory or the file. exec or no exec, an attacker can do whatever they want if they can modify memory or the file.

[chan] bitmessage <<Ext>>
Feb 13 00:04 [raw]

from argparse import ArgumentParser, FileType from collections import namedtuple from copy_reg import _reconstructor Peer = namedtuple('Peer', ('host', 'port')) safe_class_map = { 'copy_reg._reconstructor': _reconstructor, '__builtin__.tuple': tuple, 'state.Peer': Peer, } def get_class(module, name): fullname = '%s.%s' % (module, name) klass = safe_class_map.get(fullname, None) if not klass: raise UnpicklingError('Unpickling %s not supported' % fullname) return klass parser = ArgumentParser(description='Remove non-onion entries from knownnodes') parser.add_argument('input', nargs='?', default='-', type=FileType('rb'), help='Path of knownnodes.dat to filter or - for stdin (default: stdin)', metavar='in_knownnodes') parser.add_argument('output', nargs='?', default='-', type=FileType('wb'), help='Path to save filtered knownnodes or - for stdout (default: stdout)', metavar='out_knownnodes') args = parser.parse_args() try: import cPickle as pickle unpickler = pickle.Unpickler(args.input) unpickler.find_global = get_class except ImportError: import pickle class Unpickler(pickle.Unpickler): def find_class(self, module, name): return get_class(module, name) unpickler = Unpickler(args.input) streams = {} for stream, nodes in unpickler.load().iteritems(): streams[stream] = {peer: attrs for peer, attrs in nodes.iteritems() if peer.host[-6:] == '.onion'} args.output.write(pickle.dumps(streams).replace('__main__', 'state', 1))

[chan] bitmessage
Feb 13 01:03 [raw]

slick work

[chan] bitmessage
BM-2cWy7cvHoq3f1rYMerRJp8PT653jjSuEdY

Subject Last Count
PyBitmessage Security Scan on Branch v0.6 May 26 08:07 9
YOU WANNA HIRE A LEGIT HACKER????? May 26 04:39 5
Security Test on PyBitmessage Branch Master May 26 00:11 1
#2 May 25 22:41 6
minimum difficulty for chans May 25 16:45 16
BM-2cWkFSxB4cyeNVr99tgJdkMA2nfivbXLiH May 25 07:07 2
ein kleines pyBM Nebenproblem in KDE LiquidShell May 24 18:14 1
PyBitmessage 0.6.3.2 blacklist whitelist May 24 06:29 6
Test DML May 24 02:18 1
Now, following my own advice, adding channel bitmessage and general to the blacklist May 23 15:50 9
hyperboria node [fc5b:acf7:9762:439c:394d:02bb:d603:05de]:8444 May 23 01:34 3
Feature request: delete all messages from user May 22 10:46 2
(no subject) May 22 06:46 7
Github Wiki complaint May 21 08:49 12
EFAIL?! May 21 08:25 26
ERROR - Error Processing May 21 08:25 3
Curious May 21 02:17 32
Is bitmessage within whonix bad? May 20 21:24 14
Duplicate messages May 20 21:08 1
Download of Windows binary from Bitmessage.org May 20 07:25 3
How to create a "send only" bitmessage address May 20 04:35 1
/join #bitmessage on eris.us.ircnet.net :6667 May 19 21:46 3
hey - why not make pyBM as shitty as "Signal-App" by Marlinspike ? May 19 20:30 7
use Claws mail-App with pyBM and python May 19 20:28 5
A question May 18 23:24 2
A Few Bitmessage Internals for New Users May 18 23:08 5
May 18 17:33 1
Ideas for countering trolls and spam May 18 12:54 98
DARKNET DIRECTORY ASSISTANCE May 18 02:25 1
Broadcast messages May 17 23:24 24
2018 : Der junge Karl Marx -- youtube.com/watch?v=AbM76KUm4IM -- 2 hours "Le Jeune Karl Marx" May 17 20:24 1
Signal-App is complete shit May 17 20:24 13
May 17 19:49 2
OTR interception May 17 18:00 3
auto renew one's canary using broadcast or [chan] ? May 17 10:51 1
latest in the spy world May 16 14:14 3
Curious -- GUIfied pyBM-CLI May 16 13:47 1
efail vulns May 16 13:21 1
how does the namecoin feature work? May 16 07:24 3
Email campaign to promote Bitmessage? May 15 18:09 1
NSA doesn't joke, folks May 14 23:26 2
Beaker May 14 19:27 1
Bitmessage Bug - Re: Now, following my own advice, adding channel bitmessage and general to the blacklist May 14 16:21 3
Ideas for countering trolls and spam - technology. May 14 16:21 9
BITMESSAGE May 14 14:58 2
BM in firejail May 14 14:24 1
Team Revenge May 14 09:54 1
What are these messages? May 13 07:57 8
Bitmessage Bug? May 10 19:59 1
TOR -> VPN -> TOR May 10 14:57 2
Bitmessage on Raspi May 10 09:32 2
Bloom Filter for Routing May 10 09:04 1
Alternative treatment of Bitmessage addresses for use as public channels May 9 16:12 4
deterministic passphrases May 8 16:54 21
nothing wrong with suicide these days May 8 10:30 2
What's Peter Todd's public key? May 8 10:27 7
BMinstallMenu - easy download + run Bitmessage from py source in one single menu May 8 08:46 1
BMinstallMenu - easy download + run Bitmessage from py source in one single menu May 7 18:38 2
Why there are so many alternative Bitmessage implementations? May 7 18:31 14
modding pyBM May 7 18:17 4
bm hidden service settings May 7 10:48 1
bitmessage feature proposal May 7 10:38 1
This shit world May 7 07:22 2
Outgoing connections May 7 04:53 2
"time to live" ? May 7 03:27 2
OTR on Bitmessage May 7 02:06 31
Newbies! READ ME! (Bitmessage Primer) May 7 00:43 1
For Bitmessage Devs - GUI Interface Design May 6 23:18 1
O M E G A May 6 19:14 14
Bitmessage being sandbagged? May 6 05:55 3
Is Peter Surda around? Why stop signing technical messages? May 5 22:40 3
How to decrypt past objects? May 5 08:18 14
PyBM Error - no sufficient space in / partition but /home have lot's of free space May 4 13:42 3
Anybody seen this error before? May 4 12:58 4
<h1>HTML tags are enabled in subject tooltips</h1> May 3 22:17 3
is that right? May 3 07:33 6
RE: pyinstaller binaries do not run May 2 07:37 1
RE: hidden chan? May 1 06:05 1
hidden chan? Apr 30 16:15 2
bitmessage takes long to connect and finds only few peers Apr 29 10:54 2
pyinstaller binaries do not run Apr 29 09:43 1
ready-made Linux distro with BM included via TOR : "Merlot" Apr 29 09:27 1