dammIT - conferenceshttps://dammit.nl/2017-08-07T15:28:00+02:00A rantboxEuroPython 2017, part 22017-08-07T15:28:00+02:002017-08-07T15:28:00+02:00Michiel Scholtentag:dammit.nl,2017-08-07:/europython2017-part2.html<p><a href="https://shuttereye.org/goingout/2017_europython/IMG_20170711_182348.jpg/view/"><img alt="IMG_20170711_182348" src="https://shuttereye.org/images/0f/0f1332a16d8f3130_2000-2000.jpg"></a></p>
<p>Photos of the whole <a href="https://ep2017.europython.eu/en/">EuroPython 2017</a> conference are available at <a href="https://www.flickr.com/groups/3845891@N22/">the EuroPython 2017 flickr group</a>, the <a href="https://ep2017.europython.eu/en/europython/photos/">EuroPython 2017 website</a>, and <a href="https://shuttereye.org/goingout/2017_europython/">my photo gallery</a>.</p>
<p>Videos (lifestream captures for now) are at <a href="https://www.youtube.com/playlist?list=PL8uoeex94UhEP1C94Fgdn3PdXitS8RtOV">YouTube</a>, more will go live later at the <a href="https://www.youtube.com/c/EuroPythonConference">EuroPython YouTube Channel</a>. Recommended.</p>
<h2>Day three, Wednesday</h2>
<p>We kicked off with the …</p><p><a href="https://shuttereye.org/goingout/2017_europython/IMG_20170711_182348.jpg/view/"><img alt="IMG_20170711_182348" src="https://shuttereye.org/images/0f/0f1332a16d8f3130_2000-2000.jpg"></a></p>
<p>Photos of the whole <a href="https://ep2017.europython.eu/en/">EuroPython 2017</a> conference are available at <a href="https://www.flickr.com/groups/3845891@N22/">the EuroPython 2017 flickr group</a>, the <a href="https://ep2017.europython.eu/en/europython/photos/">EuroPython 2017 website</a>, and <a href="https://shuttereye.org/goingout/2017_europython/">my photo gallery</a>.</p>
<p>Videos (lifestream captures for now) are at <a href="https://www.youtube.com/playlist?list=PL8uoeex94UhEP1C94Fgdn3PdXitS8RtOV">YouTube</a>, more will go live later at the <a href="https://www.youtube.com/c/EuroPythonConference">EuroPython YouTube Channel</a>. Recommended.</p>
<h2>Day three, Wednesday</h2>
<p>We kicked off with the keynote <a href="https://ep2017.europython.eu/conference/talks/if-ethics-is-not-none">'If Ethics is not None'</a> by Katharine Jarmul with a lot of food for thought.</p>
<p><a href="https://shuttereye.org/goingout/2017_europython/IMG_20170712_094331.jpg/view/"><img alt="Expert system spoof on 'do you know where your children are?'" src="https://shuttereye.org/images/47/47594d4d25361f1f_2000-2000.jpg"></a></p>
<p><a href="https://dammit.nl/computer-ethics-quotes.html">Some good quotes were used</a>, and the book <a href="https://books.google.nl/books?id=NnM-uISyywAC&pg=PA27&dq=wiener+It+may+very+well+be+a+good+thing+for+humanity&sa=X&redir_esc=y#v=onepage&q=wiener%20It%20may%20very%20well%20be%20a%20good%20thing%20for%20humanity&f=false">Cybernetics or Control and Communication in the Animal and the Machine</a> by Norbert Wiener seems like a good read. As community, we should take care to keep thinking about the implications of the technology we create, as we should not blindly trust it or its users.</p>
<p>Next up the <a href="https://ep2017.europython.eu/conference/talks/mary-had-a-little-lambda">'Mary had a little lambda' talk</a> was a really fun (re)introduction to lambda calculus. Yes, the calculus, not the lambda functions in Python, but Anjana Vakil continued with implementing the Church Numerals in lambda functions and we did some arithmetic with Python in Church Encoding. Fun and energetic talk, which I would have loved to have at university, instead of the long dry one I got there on the same material. <a href="https://en.wikipedia.org/wiki/Lambda_calculus">Lambda calculus</a> was invented by Alonzo Church, starting in 1932, for 'expressing computation based on function abstraction and application using variable binding and substitution' and is Turing complete.</p>
<p>Lee Sheng's talk on <a href="https://ep2017.europython.eu/conference/talks/teeing-up-python-code-golf">'Teeing up Python: Code Golf'</a> (check out the slides there) had a lot of good points about writing sane, short and readable code. For example, try using a default: <code>to_mail = my_contact.get("address", "UNKNOWN")</code> instead of checking if a key exists and then getting it from the list. See the slides on why this is less strokes (less assignments, lookups and more). Automatically cleaning up resources by using the (well-known, right?) <code>with open('myfile', 'r') as infile:</code> and such also make for less error-prone code.</p>
<p>A (list) comprehension also safes you some strokes, and can result in better readable code:</p>
<pre><code>result = []
for item in things:
if condition(item):
result.append(transform(item))
# 14 strokes
result = [ transform(item) for item in things if condition(item) ]
# 12 strokes
</code></pre>
<p>In <a href="https://docs.google.com/presentation/d/e/2PACX-1vQ5ymUw7j58oEU0pTsTL-OUNMNrTq_HUE8_Z-tsKgFPo8FwNfK_EgWmUtn0oEG11MCM_259_aMmLdSY/pub?start=false&loop=false&delayms=3000#slide=id.p">the slides</a> (<a href="https://ep2017.europython.eu/media/conference/slides/teeing-up-python-code-golf.pdf">PDF</a>) are some more tips.</p>
<!--
Even the speakers were nerding out:
[![TARDIS speaker](https://shuttereye.org/images/7e/7efd3717937e9fad_2000-2000.jpg)](https://shuttereye.org/goingout/2017_europython/IMG_20170711_092517.jpg/view/)
-->
<p>An interesting Big Data/distributed computing talk was <a href="https://ep2017.europython.eu/conference/talks/realtime-distributed-computing-at-scale-in-pure-python-storm-and-streamparse">Realtime Distributed Computing At Scale (in pure Python!): Storm And Streamparse</a> by Alexander Lourenco from Parse.ly.</p>
<p>Storm is a distributed real-time computation system, which simplifies workers and queues. Streamparse is basically Pythonic Storm; Nimbus and Storm UI are visual frontends.</p>
<p>You can install the Storm environment from the Apache Storm site.</p>
<p>Streamparse is even easier, just use pip:</p>
<pre><code>pip install streamparse
sparse quickstart
sparse run
</code></pre>
<p>The <a href="https://ep2017.europython.eu/conference/talks/infrastructure-design-patterns-with-python-buildbot-and-linux-containers">Infrastructure design patterns with Python, Buildbot, and Linux Containers</a> talk was a bit disappointing. It was basically David Liu's hobby project of (ab)using Dask (a distributed task system) and buildbot (normally known to be a continuous integration (CI) framework) for things it is not really intended, like remotely executing random scripts.</p>
<p>I better could have gone to <a href="https://ep2017.europython.eu/conference/talks/how-to-make-money-with-your-python-open-source-project">How to make money with your Python Open-Source Project</a> by Max Tepkeev, which would have been relevant for my own projects and probably for some of Sanoma's efforts too.</p>
<p><a href="https://shuttereye.org/goingout/2017_europython/IMG_7069_crop.jpg/view/"><img alt="Roman bridge" src="https://shuttereye.org/images/ce/ce5a3519d519279d_2000-2000.jpg"></a></p>
<h2>Day four, Thursday</h2>
<p>Kicking off with <a href="https://ep2017.europython.eu/conference/talks/type-annotations-in-python-3-whats-whys-wows">Type Annotations in Python 3: Whats, whys & wows!</a> by Andreas Dewes, Thursday was in for a good line-up.</p>
<p>Executive summary: Python 3.6 is great, start using it. However, in older 3.x versions and also in 2.7, some nifty things can be done with hints about variable types.</p>
<ul>
<li>Gradual typing, where you add annotations to code where it makes sense (so you don't have to rewrite your complete codebase)</li>
<li>Function annotations were the first step in Python 3.0 in 2006 (PEP 3107)</li>
<li>In 3.5 Type Hints were introduced (including the theory and such)</li>
<li>Python interpreter stores those annotations in <code>__annotations__</code> and ignores it otherwise</li>
<li>External tools like <a href="http://mypy-lang.org/"><em>mypy</em></a> and <em>pycharm</em> do the heavy lifting</li>
<li>Deliberate choice to do it like this, so it can evolve; eventually we might want to have internal tools</li>
<li>We lose Python 2 compatibility though, but then we can use <em>type comments</em>, which are backported to 2.7</li>
<li>A third approach is stub files, ending with .pyi, which <em>mypy</em> will also look at, ignoring the .py file for type checking. This way, we can add only the annotations there. Feels a bit like writing a header file</li>
<li>Check the documentation of the typing module for more info, lots of usecases (new in 3.5)</li>
<li>The <a href="https://www.slideshare.net/japh44/type-annotations-in-python-whats-whys-and-wows">slides are on Slideshare</a></li>
</ul>
<p><a href="https://ep2017.europython.eu/conference/talks/inside-airbnb-visualizing-data-that-includes-geographic-locations">Inside Airbnb: Visualizing data that includes geographic locations</a> by Guillem Duran Ballester provided some interesting visualisations of Airbnb locations (specifically Barcelona and Mallorca). By doing so, he introduced quite some useful tooling for drawing maps, drawing shapes on them (for example for area's to aggregate datapoints to densities) and various shaders. Everything was demoed through <a href="https://github.com/Guillem-db/Inside-Airbnb-EP17">Jupyter notebooks</a>. Data from <a href="http://insideairbnb.com/">insideairbnb</a> was used.</p>
<p>Some take-aways:</p>
<ul>
<li><a href="https://github.com/bokeh/bokeh">bokeh</a> is a great library to work with Google Maps (take a look at lat & lon, zoom and map_type)</li>
<li><a href="https://github.com/HCsoft-RD/shaolin">shaolin</a> for colormaps</li>
<li>shapefiles with <a href="https://github.com/Toblerity/Shapely">shapely</a> (for drawing overlays on the map, for example to colour an area of a town)</li>
<li>holoviews and geoviews (matplotlib, bokeh and shapely as backends)</li>
<li><a href="https://github.com/bokeh/datashader">datashader</a>: plotting big data made easy; lots of Jupyter notebooks as examples</li>
<li><code>dynamic=True</code> will recalculate the bins that datashader makes, making it nicer to look at when zooming in (with dynspread/datashade)</li>
<li>one of the downsides of datashader is that when tweaking, it errors until you get everything right (for example in Jupyter)</li>
<li><a href="http://geo.holoviews.org/Working_with_Bokeh.html">Working with Bokeh documentation</a></li>
<li>integrating openstreetmaps with bokeh should be possible, speaker needs to look into it (but Google Maps is default)</li>
<li><a href="https://github.com/dsolanno/BarcelonaRentsStatus">The code of the rent status of Airbnb venues</a>.</li>
</ul>
<p><a href="https://shuttereye.org/goingout/2017_europython/IMG_20170712_170046.jpg/view/"><img alt="Conference location artifact" src="https://shuttereye.org/images/61/613362cc8eb23616_2000-2000.jpg"></a></p>
<p>The <a href="https://ep2017.europython.eu/conference/talks/an-introduction-to-pytorch-autograd">An introduction to PyTorch & Autograd</a> talk from Paul O'Grady provided some tools to do deep learning with Python.</p>
<p>Tensor (ndarray) operations on the GPU (instead of being constrained to CPUs) are a lot faster.</p>
<p><code>pytorch</code> is the new kid on the block. It follows Lua torch, with same underlying C libraries and is a define-by-run framework as opposed to define-and-run, leads to dynamic computation graphs, looks more Pythonic. The native data format is <a href="https://en.wikipedia.org/wiki/Tensor">tensors</a>, which is a bit different than what we are used to, but makes sense in these computations. Torch supports in-place adds and such (.add_() ) and plays well with numpy; it can bridge back and forth (for example having matrix operations and getting a tensor back). To get more insight in the data, you can reshape tensors using views. One of the nice optimalisation things is that tensor computation can be moved to and from GPU (CUDA). Support for variables is added through the <a href="http://pytorch.org/docs/master/autograd.html">torch.autograd package</a>.</p>
<p>Next up was <a href="https://ep2017.europython.eu/conference/talks/developing-elegant-workflows-in-python-code-with-apache-airflow">Developing elegant workflows in Python code with Apache Airflow</a> where Michał Karzyński talked about creating data flows with Airflow.</p>
<p>The definition of a workflow is a sequence of tasks, started on a schedule or triggered by an event, frequently used to handle big data processing pipelines.</p>
<p>Apache Airflow is Open Source, based on Flask, using Celery.</p>
<p>A flow is a Directed Acyclic Graph (DAG). An Operator is a single task, which can be retried automatically, should be idempotent and is a Python task with an execute method. Airflow uses xcom as a means of communication between task instances; a task is saved in the database as a pickled object; it is best suited for small objects. An introductory Airflow tutorial can be found on the speaker's weblog: <a href="http://michal.karzynski.pl/blog/2017/03/19/developing-workflows-with-apache-airflow/">Developing workflows with Apache Airflows</a>.</p>
<p><a href="https://ep2017.europython.eu/conference/talks/inspect-or-gadget">Inspect (Or Gadget)</a> by Hugues Lerebours and Renaud Bauvin introduced a really useful library: <a href="https://docs.python.org/3/library/inspect.html"><code>inspect</code></a> which provides quite some extra functions apart from the already useful Python buildins. The demo consisted of a piece of code that checks if docstrings are up-to-date, also validating against documented types. Good material to check out when needing to do some introspection.</p>
<p>Another talk about improving your code's quality was <a href="https://ep2017.europython.eu/conference/talks/fixture-factories-for-faster-end-to-end-tests">Fixture factories for faster end-to-end tests</a> by Stephan Jaensch, speaking about creating better fixtures to improve tests in this age of microservices and other loosely coupled services.</p>
<p>When testing such a setup, you want end-to-end integration, replicating production as much as possible (above 'regular' integration tests). This results in slow, most expensive tests. Tools used in such a test setup are <code>pyramid</code>, <code>swagger</code>, <code>openapi</code>, and <code>sqlalchemy</code>. Creating it needs lots of db setup on all the components, lots of SQL scripts, and is hard to write, and maintain.</p>
<p>Taking inspiration from Django, fixture factories are introduced. They allow you to create fixtures, not only for your service, but for downstream services as well and without code duplication. These fixtures take care of common default values and make sure data is logically correct. They integrate nicely with <code>pytest</code>.</p>
<p>Why not use models? No foreign key (FK) checking used in this setup, so cannot use the PK and FK's.</p>
<p>This setup helps with test repeatability, as it eliminates dependability between tests (order). Use fixture factories for faster development and more correct test data; convert tests for test isolation and repeatability, take advantage of it by executing tests in parallel. A downside is that it is potentially slower (as you are not sharing data across tests).</p>
<p><a href="https://github.com/sjaensch/faster_end_to_end_tests_talk">Slides</a> (or <a href="https://ep2017.europython.eu/media/conference/slides/fixture-factories-for-faster-end-to-end-tests.pdf">PDF</a>).</p>
<p><a href="https://shuttereye.org/goingout/2017_europython/IMG_6966.jpg/view/"><img alt="Rimini beach at night with the ferris wheel" src="https://shuttereye.org/images/01/01000b9b07c7898a_2000-2000.jpg"></a></p>
<h2>Day five, Friday</h2>
<p>Last day started with <a href="https://ep2017.europython.eu/conference/talks/finding-bugs-for-free-the-magic-of-static-analysis">Finding bugs for free: The magic of static analysis</a> in which Mark Shannon talked about code analysis: finding facts about your code, without running it.</p>
<p>When you get to maintain a code base, you want to know where are the bugs, is this expensive to maintain; things you preferably want to get quantified by an automated code review.</p>
<p>What makes for good code analysis? It has to be flexible: can you extend it? Also, accurate: don't waste users time and insightful: find non-trivial things. This can be done for Python, but it's harder than for a statically-typed language, but can be more valuable as well.</p>
<p>So, what makes <a href="https://lgtm.com/">lgtm</a> (automated code reviews service) flexible? It provides an object-oriented query language, in which queries can be simple, yet powerful. Also, a powerful library is provided.</p>
<p>Code is parsed into an abstract syntax tree (in contrast to concrete syntax tree), from which a control flow graph (CFG) is generated. By doing CFG splitting, you can deduct faulty code paths and drill down to bugs.</p>
<p>Next up was <a href="https://ep2017.europython.eu/conference/talks/practical-debugging-tips-tricks-and-ways-to-think">Practical Debugging - Tips, Tricks and Ways to think</a> by Radoslav Georgiev, speaking about patterns and how stack traces are your friend.</p>
<p>He had a set of rules to live by as a developer, starting with the golden rule: if you find a bug, add a test for it (making it reproducible). Keep in mind tostay away from constant regression, and when debugging use ipdb/pdb (better than print of course) (by the way, <code>launch_ipdb_on_exception</code> as context manager, saves you a lot of 'cont, enter, cont, enter'), check tests, check input validation, check algorithms, check the system design and of course check your understanding of the problem space.</p>
<p>Another practical tip was to explain your problem to someone (the well-known rubber duck debugging).</p>
<p>He also proposed to do parallel debugging:</p>
<ol>
<li>ask a question in StackOverflow</li>
<li>open GitHub issue</li>
<li>ask a co-worker for help</li>
<li>all of that while you are still debugging</li>
</ol>
<p>However, I think that's not really a nice thing to do, because you are offloading your problem (possibly generating a lot of noise in the process). So, contribute your solutions back to the community by answering issues on StackOverlfow, writing on GitHub issues, opening PRs with bugfixes and examples, submitting documentation improvements, and... do a talk on a conference :)</p>
<ul>
<li><a href="https://github.com/RadoRado/EuroPython2017">https://github.com/RadoRado/EuroPython2017</a></li>
<li><a href="https://github.com/HackSoftware/">https://github.com/HackSoftware/</a></li>
<li><a href="https://hacksoft.io">hacksoft.io</a></li>
</ul>
<p>The 'Magic considered harmful' talk was cancelled, but there is <a href="https://www.youtube.com/watch?v=ZLfDpDG2fcU">this recording of a previous version</a>.</p>
<p><a href="https://ep2017.europython.eu/conference/talks/lessons-learned-in-x-years-of-parallel-programming">Lessons learned in X years of parallel programming</a> in which Michele Simionato told about lessons learned solving big problems.</p>
<p>Embarassingly parallel problems are common and may be more easy to solve than you think. It's generally not trivial though, which is to be expected. I recommend taking a look at <a href="https://ep2017.europython.eu/media/conference/slides/lessons-learned-in-x-years-of-parallel-programming.html">his (plain text) slides</a>.</p>
<p>Fergal Walsh had some intesting thoughts in <a href="https://ep2017.europython.eu/conference/talks/rethinking-how-we-build-http-apis">Rethinking how we build HTTP APIs</a>, in which he asked why API's need to be so rigid and REST, for example. One of his projects is <a href="https://github.com/fergalwalsh/pico">pico</a>, which is a really tiny framework to help you quickly create an API from a codebase without having to worry about correctness and such (REST).</p>
<p>It enables for easy interactive development (no need to set up all kinds of environment when you want to want to use ipython when developing for example). This is done by using decorators to fill function parameters that pico needs; this makes it possible to use and test pico from ipython and such.</p>
<p>Speaking about decorators, <code>wrapt</code> is a nice decorator library.</p>
<p>A great talk about the social aspects of technology an in IT specifically was <a href="https://ep2017.europython.eu/conference/talks/overcoming-cognitive-bias">Overcoming Cognitive Bias</a> by Anna Martelli Ravenscroft (wife of Alex Martelli, writer of Python in a Nutshell; both authored Python Cookbook).</p>
<p>This is a subject that still needs a lot of attention and was a really good talk to raise awareness. If you ask 'are you a programmer?' to a woman at a conference, that's cognitive bias at work.</p>
<p>For more information, see this video: <a href="http://pyvideo.org/pycon-us-2010/pycon-2010--diversity-as-a-dependency---49.html">Diversity as a dependency</a> and watch the <a href="https://ep2017.europython.eu/conference/talks/despicable-machines-how-computers-can-be-assholes">"Despicable machines: how computers can be assholes" talk</a>.</p>
<p>The lightning talks had a few interesting titbits:</p>
<ul>
<li><a href="http://bit.ly/talk-to-jupyter">A way of interacting with Jupyter notebooks</a>.</li>
<li>Run Python with warnings enabled. This tells about unclosed files and a lot more things. To do so, run <code>python3 -Wd -b</code> or for version 2: <code>python2 -Wd -t -3</code>.</li>
<li>Some pointers when giving talks: structure beats slides (have 3-6 clear points); have a story; take-away value beats completeness (audience won't know what you left out anyway); speech_projects</li>
<li>http://tinyurl.com/PerceptionMusic</li>
<li>ssim for parsing flight schedule strings (in .sir files) to something usable: <code>pip install ssim</code></li>
<li>pyjok.es: <code>pip install pyjokes</code></li>
<li><a href="https://github.com/mariocj89/dothub">Manage GitHub by API instead of as a mere human with dothub</a>.</li>
</ul>
<p><a href="https://shuttereye.org/goingout/2017_europython/IMG_7060.jpg/view/"><img alt="Rimini yacht harbour" src="https://shuttereye.org/images/5f/5ff8861f3c1c164f_2000-2000.jpg"></a></p>EuroPython 2017, part 12017-07-13T10:11:00+02:002017-07-13T10:10:00+02:00Michiel Scholtentag:dammit.nl,2017-07-13:/europython2017-part1.html<p><img alt="The EuroPython 2017 location (PalaCongressi di Rimini (PCR)), seen from the park next to it" src="https://shuttereye.org/images/1c/1c1e0e87cba52524_2000-2000.jpg"></p>
<p><a href="https://ep2017.europython.eu/en/">EuroPython 2017</a></p>
<p>(Needs editing after I get some sleep and more time)</p>
<p>This post also appeared on <a href="https://medium.com/sanoma-technology-blog/europython-2017-part-1-4067aabc5487">the Sanoma Technology weblog</a>.</p>
<h2>Day zero, Sunday</h2>
<p>After a rather pleasant flight on Sunday morning, where out of 80 passengers, I sat next to the only other EuroPython visitor, I took a nicely …</p><p><img alt="The EuroPython 2017 location (PalaCongressi di Rimini (PCR)), seen from the park next to it" src="https://shuttereye.org/images/1c/1c1e0e87cba52524_2000-2000.jpg"></p>
<p><a href="https://ep2017.europython.eu/en/">EuroPython 2017</a></p>
<p>(Needs editing after I get some sleep and more time)</p>
<p>This post also appeared on <a href="https://medium.com/sanoma-technology-blog/europython-2017-part-1-4067aabc5487">the Sanoma Technology weblog</a>.</p>
<h2>Day zero, Sunday</h2>
<p>After a rather pleasant flight on Sunday morning, where out of 80 passengers, I sat next to the only other EuroPython visitor, I took a nicely airconditioned train from Bologna to Rimini. I really hoped that airco was everywhere, otherwise the 33C would seriously hamper my capability of even remembering what was said just five minutes ago.</p>
<p>My newfound acquaintance and me got some good Italian food inside us, cooled down for a few minutes at the beach, breathing in the sea air and explored Rimini centre for a bit, where the heat was slowly leaking out of the stones.</p>
<h2>Day one, Monday</h2>
<p>Moving on to Monday, I was thankfully doing just fine. The walk through the park (literally) to the venue was warm and lengthy but shady and beautiful, and the venue was well organised and cool, suiting my feeble brain.</p>
<p>Starting off with an enlightening keynote by Armin Ronacher, where we were reminded of the fact that files and other input still are not UTF-8 (even going beyond UTF-8 now we have emoticons, which are in an even higher byte space), making it hard to do string slicing and such. Also, don't subclass dictionaries. Just don't.</p>
<p>I moved on to a hands-on session with Django and Celery, as I wanted to see if there were some features I wasn't aware of yet, or pointers to solutions to some minor issues we are having at Sanoma.</p>
<p>Coming away with some remarks about just using chains and groups, as things like chords can act weirdly in certain circumstances (like on RabbitMQ, which surprise surprise is what we use at home), I thought that maybe some small refactoring is in order.</p>
<p>Next up was a talk about how you can know that the mocks you created for a (third party) API are valid. Vuforia (an image matching service) was used as example. <code>requests-mock</code> is of course a useful tool here, and the speaker made his mock into a Flask app, which translated requests-mock to something that could be used by their library.</p>
<p>The verifying part is done by running all the tests twice, once against the real service, and once against your mock. This checks if the responses are the same. By scheduling tests with for example Travis CI, you can then find out about (undocumented) changes to the third party service in a timely and targeted fashion (as the failing test points to the exact location), making it a lot more easy to fix your library.</p>
<p>Also, if you ship software, you really should ship it with a verified fake, so people can use that to test their own software; it adds a lot of value to your product.</p>
<p>After the lunch it was time to measure, don't guess. A nice hands-on presentation of how to profile your code, so you can optimise where needed. Premature optimisation is the root of all evil, so knowing where to actually put your efforts is really useful.</p>
<p>A string of tools were introduced, apart from the excellent Jupyter notebook software to experiment with (really, if you don't know it, or never used it, <code>pip install jupyter</code> and <code>jupyter notebook</code> it). Build into Python itself is timeit (<code>import timeit</code>, <code>timeit.default_timer</code>, don't use os_time or time_clock, as they will give different results depending on your OS platform). <code>snakeviz</code> is a browser based graphical viewer of the output of Python's cProfile module, representing time spent in various parts of the code as concentric circles, making it easy to zoom into troublesome parts. It can be run standalone, or as a module: <code>python -m snakeviz</code>, <code>python -m cProfile -o pi.stats simple_pi.py</code> for example (<a href="http://www.python-academy.com/download/europython2017/">example code</a>), or just <code>snakeviz pi.stats</code>. Be aware that tool measures itself too of course.</p>
<p>In Jupyter, you can load such modules to experiment with interactively:</p>
<pre><code>load profile_me.py <shift>-<enter>
%load_ext snakeviz
%snakeviz test()
</code></pre>
<p>Same with <code>timeit</code>:</p>
<pre><code>%timeit
%%timeit # multi-line version, where you can add more lines of code to test
%timeit? # to get help of course
</code></pre>
<p>The <a href="https://github.com/rkern/line_profiler">line profiler</a> is another tool that gives information about what lines of code are troublesome:</p>
<ul>
<li><code>pip install line_profiler</code></li>
<li><code>kernprof (-v)</code></li>
<li>line-by-line profiling: -l</li>
<li><code>kernprof -v -l profile-me_use_line_profiler.py</code></li>
<li>in Jupyter: <code>load_ext line_profiler</code></li>
</ul>
<p>For memory usage, <a href="https://pythonhosted.org/Pympler/">Pympler</a> can be used. It keeps track of changes in the size of data structures, so you can see where your code balloons, or even where memory is leaking:</p>
<pre><code>from pympler import tracker
t = tracker.SummaryTracker()
t.print_diff()
big = list(range(10**6))
t.print_diff() # shows how big the datastructure we just created is (as it shows the difference with the previous measurement)
</code></pre>
<p>With the Python functools one can create decorators (like in ./measuring/memory_size_pympler.py: measure_memory in the example code linked above). This one creates a decorator the create a measurement of a function, storing it in a variable.</p>
<p>matplotlib is another great library to visualise data.</p>
<p>Finally, there is another memory profiler:</p>
<p><code>pip install memory_profiler</code>, <code>%load_ext memory_profiler</code> and then you can do something like:</p>
<pre><code>import use_mem # an application
%mprun -f use_mem.use_mem use_mem.use_mem(numbers)
</code></pre>
<p>So, enough pointers to play with and explore from.</p>
<p>The conference seems well balanced and organised and the participants are generally really social, making for interesting chats over food.</p>
<p>It's also refreshing to see the amount of female attendees. I really hope it's a good sign for the future, as in my opinion we need more of their talent in our field of work.</p>
<h2>Day two, Tuesday</h2>
<p>Instead of walking, I decided to make use of the loaner bikes of the hotel, which was really an improvement.</p>
<p>Cooling down, the <a href="https://ep2017.europython.eu/conference/talks/how-to-create-inspiring-data-visualizations">kickoff of the day</a> was about data visualisation by Dutch visualiser Jan Willem Tulp. This was great to behold, seeing how he made big and complex datasets comprehensible and even beautiful, appealing to laymen too (his animated globe with number of trees was used in a nice demo from a news publication).</p>
<p><a href="https://shuttereye.org/goingout/2017_europython/IMG_20170711_094234.jpg/view/"><img alt="Star chart" src="https://shuttereye.org/images/9b/9bb9995b59290b0f_2000-2000.jpg"></a></p>
<p>[animation]</p>
<p>After enjoying the nice visuals and inspirational talk, I was curious how to <a href="https://ep2017.europython.eu/conference/talks/write-more-decorators-and-fewer-classes">use and write decorators</a> to eliminate the need for some classes.</p>
<p>There is a nice talk about why you should stop writing classes: https://www.youtube.com/watch?v=o9pEzgHorH0</p>
<p>Decorators are rather easy to use, not trivial to write, but also not that hard to do, and can be really helpful in making your code more clear.</p>
<p>Continuing with a <a href="https://ep2017.europython.eu/conference/talks/abstract-base-classes-a-smart-use-of-metaclasses">smart use of metaclasses by using abstract base classes</a> (ABCs). Those help with testing behaviour, not structure.</p>
<p>When trying to find out if some object is what you need, checking the type (isinstance() and such) is not really the intention, because some object or class can behave like a certain class, but is of course not exactly that implementation/type you are checking. A perfect solution would be something like: <code>if behaveslike(someobj, ListBehaviour):</code></p>
<p>Abstract base classes are categories (like labels/tags, but it's just a promise to act in a certain way, no checks are done). Using them though, enables you to do the above comparison.</p>
<p>Python Collections has a lot of classes that represent interesting behaviours (go check them).</p>
<p>Some more notes:</p>
<ul>
<li>Python is based on delegation, which is good news; magic functions come into play.</li>
<li>Virtual subclasses: ParentClass.register(ChildClass), with opposite relationship, where the ParentClass knows the ChildClass, but not the other way around like in regular subclassing. Useful for when you want to know all the subclasses of a certain class. Registering is a promise, no check done, so be careful.</li>
<li>A metaclass can put things into the class that is linked to it.</li>
<li>You can abstract base classes to build interfaces, but in the speaker's opinion this is not the way to go. Can be useful though.</li>
</ul>
<p>After that I attended a slightly confused talk about design patterns and why we don't need them in Python. The idea was good, as design patterns were thought up in a time that programming languages were in a place where developers needed them to enforce certain restrictions, but then the speaker started showing where certain design patterns were still used or usable inside Python. All in all a good way to brush up on my old knowledge of them and seeing why I don't program in Java anymore.</p>
<p>Some of the points and design patterns:</p>
<ul>
<li>Java: design patterns! (general reusable solution to a commonly occurring problem; formalised best practices; just outlines though, not ready to use)</li>
<li>Singletons in python? Modules!</li>
<li>Visitor pattern: ASTs (abstract syntax trees, linters use those)</li>
<li>Decorator patterns (not the @decorator in python)</li>
<li>Is magic worth the effort? It depends, readability wins.</li>
</ul>
<p>His take-away was good though: know your tools well, get inspiration from other languages and communities, know the business domain of your project.</p>
<p>After lunch there was a subject that I am really curious about: "There should be one obvious way to bring python into production" by Sebastian Neubauer.</p>
<p><a href="https://ep2017.europython.eu/conference/talks/there-should-be-one-obvious-way-to-bring-python-into-production">His slides</a> are available from the talk page, which are a good set of guidelines.</p>
<p>It talked about the various stages inside the delivery pipeline. When a software product has to be shipped, it has to be built and packaged. Some requirements here are that it should be built once, used everywhere, the possibility to compile it for the target systems and that versions should be really unique, preferably signed. You should not have a version 1.0.2 without $fileX and one with $fileX (because you forgot to include it in the first build).</p>
<p>Nice to have: upload the build to an artifact repository.</p>
<p>A risk is that the build environment is misconfigured.</p>
<p>Testing should be automated, with near production-like and reproducible conditions, and minimal changes should be done for testing reasons. Nice to have are of course fast feedback and having the tests run after each commit on all branches (something that we are already doing in quite some projects in Sanoma).</p>
<p>A risk here is that the tests test the test env, not production (e.g., pytest pulls in $package that turns out to be missing when deploying to production).</p>
<p>After that, Staging/QA has to receive an automated deploy in a production-like environment, with nearly no changes for testing purposes. It has the risk of being an outdated, manually maintained setup, so be sure to take care of this stage.</p>
<p>Nice to haves for a staging environment are that it is a real clone of the production system and the possibility to run A/B tests on the system.</p>
<p>Production should have all kinds of restrictions, like no compiler, no internet, health monitoring and preferably have automated deploys, automated monitoring, self-healing and automatic rolling updates and roll-backs.</p>
<p>Because of the ecosystem of software, we frequently run into the dependency hell. We have multiple layers of package managers (from OS-level, to the Python world (<code>pip</code>) and of course the JavaScript world).</p>
<p>Within the Python package management there's still much confusion around setuptools, distutils, eggs and such.
- Many outdated 'best practices' on StackOverflow etc
- pyscaffold, versioneer (templates for packages)</p>
<p>Possible solution: Nix inside a container? Someone from the audience proposed using Flatpak or Snappy instead, which sound to me better candidates, seeing how mainstream Linux distro's are already including support for them.</p>
<p>Energy started running low, so the interesting, but fast talk about <a href="https://docs.python.org/2/howto/descriptor.html">Descriptors</a> left me with some notes that I really should look into them a bit more.</p>
<p>After that, Django and GraphQL seemed like a nice break from the de-facto REST standard (not that I dislike DjangoRestFramework). This new API standard, <a href="https://facebook.github.io/react/blog/2015/02/20/introducing-relay-and-graphql.html">created by Facebook</a>, has one endpoint and uses queries instead of paths to retrieve data. This was not new for me, but I was curious to the current state of integration with Django. Turns out that using <code>graphene</code> and <code>graphene-django</code> gets you going quite nicely. Enable the app and then it's only adding some glue to be able to retrieve data from your models.</p>
<p>Authentication is handled, as is usage/viewing permission per field. Apparently there are still issues when also using DjangoRestFramework, but a pull request is in review.</p>
<p>Good to know that you should limit the nesting you can have, as queries can get rather deep otherwise. GitHub does something like that already.</p>
<p>Frontends that can be used in concert with your shiny GraphQL are Facebook's Relay for React, and the really <a href="https://github.com/apollographql">Open Source Apollo</a>.</p>
<h3>Lightning talks</h3>
<p>Reporting security issues: if someone reports to you, take it seriously. Create a howto in your README too.</p>
<p>If reporting yourself, do it privately, you might take someone off guard, also possibly endanger users.</p>
<p>Don't document your experiences for others, experience your vacations and such for yourself (you can still take pictures and souvenirs, but do it for yourself).</p>
<p>gitmate.io
gitmate-bot
coala
IGitt (an exclamation of disgust)
Interface for Git{Hub|Lab}
https://gitlab.com/gitmate/open-source/IGitt</p>
<p>Quickly create/publish packages on pypi: flit
https://pypi.python.org/pypi/flit
https://pypi.org/</p>