Python
Changes
2014+

Latest content revision: January 2022

Motivation

(Structural) pattern matching syntax is found in many languages, from Haskell,
Erlang and Scala to Elixir and Ruby. (A proposal for JavaScript is also under 
consideration.) [...]

More Than One Way To Do It

  Okay, the Zen doesn't say that there should be Only One Way To Do It. 
  But it does have a prohibition against allowing "more than one way to do it".

Response

  There is no such prohibition.  The "Zen of Python" merely expresses a preference 
  for "only one obvious way":

    There should be one-- and preferably only one --obvious way to do it. [...]

  In practice, this preference for "only one way" is frequently violated in Python. [...]

  We should not be too strict about rejecting useful functionality because it 
  violates "only one way".

$ python3
Python 3.7.4 (default, Jul 28 2019, 22:33:35)
>>> from imp import reload
__main__:1: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
>>> from importlib import reload
>>>

>>> def myzip(*args):
...     iters = list(map(iter, args))           # make iters reiterable
...     while iters:                            # guarantee >=1, and force looping
...         res = [next(i) for i in iters]      # any empty? StopIteration == return
...         yield tuple(res)                    # else return result and suspend state                  
...                                             # exit: implied return => StopIteration
>>> list(myzip((1, 2, 3), (4, 5, 6)))
[(1, 4), (2, 5), (3, 6)]
>>> [x for x in myzip((1, 2, 3), (4, 5, 6))]
[(1, 4), (2, 5), (3, 6)]

>>> def myzip(*args):
...     iters = list(map(iter, args))
...     while iters:
...         try:
...             res = [next(i) for i in iters]
...         except StopIteration:               # StopIteration won't propagate in 3.7+
...             return                          # how generators should exit in 3.7+
...         yield tuple(res)                    # but exit still == return => StopIteration
... 
>>> list(myzip((1, 2, 3), (4, 5, 6)))
[(1, 4), (2, 5), (3, 6)]
>>> [x for x in myzip((1, 2, 3), (4, 5, 6))]
[(1, 4), (2, 5), (3, 6)]

>>> list(myzip((1,2,3), (4,5,6)))
Traceback (most recent call last):
...
StopIteration

The above exception was the direct cause of the
following exception:

Traceback (most recent call last):
...
RuntimeError: generator raised StopIteration

>>> def myzip(*args):
...     iters = list(map(iter, args))
...     while iters:
...         try:
...             res = [next(i) for i in iters]
...         except StopIteration:
...             raise StopIteration             # this also fails: return required
...         yield tuple(res)                    # even though return raises StopIteration!

...             res = [next(i) for i in iters]        # StopIteration
...             res = list(next(i) for i in iters)    # RuntimeError (!)

>>> spam = 'SPAM'
>>> items = [1, 2, 3, 4]
>>> intvalue = 1023

>>> f'we get {spam} alot.'                # uses variable 'spam' in this scope
'we get SPAM alot.'

>>> f'size of items: {len(items)}'        # ditto, but 'items' and an expression
'size of items: 4'

>>> f'result = {intvalue:#06x} in hex'    # formatting syntax is allowed here too
'result = 0x03ff in hex'

>>> 'we get %s alot.' % spam              # traditional formatting equivalents
'we get SPAM alot.'

>>> 'size of items: %d' % len(items)
'size of items: 4'

>>> 'result = 0x%04x in hex' % intvalue   # '%s' % hex(intvalue) works too
'result = 0x03ff in hex'

>>> f'a manual dict: {{k: v for (k, v) in (("a", spam), ("b", intvalue))}}'    # hmm...
"a manual dict: {'a': 'SPAM', 'b': 1023}"

>>> f'a manual dict: {{k: v for (k, v) in (('a', spam), ('b', intvalue))}}'
SyntaxError: invalid syntax

>>> f'a manual dict: {{k: v for (k, v) in (("a", spam), (\'b\', intvalue))}}'  # flat is better
SyntaxError: f-string expression part cannot include a backslash

'we get %s alot' % spam          # the original expression 

'we get {} alot'.format(spam)    # the later method

f'we get {spam} alot'            # yet another way to do the same thing...

>>> name = 'Sue'
>>> age  = 53                                            # keys/values in vars()
>>> jobs = ['dev', 'mgr']                                

>>> '%(name)s is %(age)s and does %(jobs)s' % vars()     # expression: original
"Sue is 53 and does ['dev', 'mgr']"

>>> '{name} is {age} and does {jobs}'.format(**vars())   # method: later addition
"Sue is 53 and does ['dev', 'mgr']"

>>> from string import Template                          # Template: wordier option
>>> t = Template('$name is $age and does $jobs')
>>> t.substitute(vars())
"Sue is 53 and does ['dev', 'mgr']"

>>> f'{name} is {age} and does {jobs}'                   # do we really need a 4th?
"Sue is 53 and does ['dev', 'mgr']"

>>> '%s is %s and does %s' % (name, age, jobs)           # simpler is still better
"Sue is 53 and does ['dev', 'mgr']"

The py.exe launcher, when used interactively, no longer prefers Python 2
over Python 3 when the user doesn’t specify a version (via command line 
arguments or a config file).  Handling of shebang lines remains unchanged
- “python” refers to Python 2 in that case.

2018-05-16 11:21:36.290 Python[501:12731] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: 'File types array cannot be empty'
...details cut...
libc++abi.dylib: terminating with uncaught exception of type NSException
Abort trap: 6

>>> x = 9_999_998             # your number with "_"s for digit groupings
>>> x                         # but displays drop the groupings anyhow
9999998
>>> x + 1                     # ditto for derived computation results
9999999
>>> '{:,}'.format(x + 1)      # and even if you request them, it's commas
'9,999,999'                   # not very symmetric, that

>>> 99_9                      # no position-error checking provided
999
>>> 1_23_456_7890             # err, what?
1234567890

>>> _9
NameError: name '_9' is not defined
>>> 9_
SyntaxError: invalid token
>>> 9_9__9
SyntaxError: invalid token
>>> 9_9_9                     # syntax oddities checked, semantics not
999

>>> 1_234_567.99               # floating points: thousands grouping
1234567.99
>>> 1_234_567.987_654_3        # anywhere after the "." too
1234567.9876543
>>> 3.1_415_9e+100             # but anything goes, and "_"s are discarded
3.14159e+100

>>> 0b1111_1111_1111_1111      # binary integers: hex groupings
65535
>>> 0b111_111_111_111_111_1    # octal groupings, more or less?
65535

>>> 0xf_ff_fff_f_f             # hex integers: anything goes here too 
4294967295
>>> hex(0xf_ff_fff_f_f)        # and Python won't retain your "_"s
'0xffffffff'

>>> x = 9_999_998              # underscores are temporary in-code notation
>>> x                          # prints display integers as decimal digits
9999998
>>> '{:,}'.format(x)           # formats transpose numeric values on demand
'9,999,998'
>>> bin(x)                     # but numbers are always stored in binary form,
'0b100110001001011001111110'   # like this (or something a bit more complex)

>>> int('1_234_567')           # works in text read from data files too
1234567
>>> eval('1_234_567')          # though subject to the syntax rules above
1234567
>>> float('1_2_34.567_8_90')   # and does raw-data readability matter? 
1234.56789

C:\Code> py -3.5
>>> [1, 2] @ [3, 4]
Traceback (most recent call last):
  File "", line 1, in 
TypeError: unsupported operand type(s) for @: 'list' and 'list'

>>> [1, 2] @ 3
Traceback (most recent call last):
  File "", line 1, in 
TypeError: unsupported operand type(s) for @: 'list' and 'int'

>>> [1, 2] * 3
[1, 2, 1, 2, 1, 2]     # this is still repetition, not multiplication

C:\Code> py -3
>>> tbdlist = [...] * 100
>>> def tbdfunc():
        ...
>>>
>>> tbdlist[-1]        # it's a placeholder object in 3.X
Ellipsis
>>> tbdfunc()          # it's a no-op statement in 3.X (see p390 in LP5E)
>>>

C:\Code> py -3.3
>>> 'a %s parrot' % 'dead'             # for str in all 3.X: decoded Unicode text
'a dead parrot'                        # but not for bytes: text encoding unknown 

>>> b'a %s parrot' % 'dead'
TypeError: unsupported operand type(s) for %: 'bytes' and 'str'

>>> b'a %s parrot' % b'dead'
TypeError: unsupported operand type(s) for %: 'bytes' and 'bytes'

C:\Code> py -3.5
>>> b'a %s parrot' % b'dead'                  # new in 3.5: bytes, %s == %b
b'a dead parrot'

>>> b'a %s parrot' % bytes([0xFF, 0xFE])      # works for non-ASCII bytes too
b'a \xff\xfe parrot'

>>> b'a %s parrot' % 'dead'                   # but %s (%b) allows bytes only!
TypeError: %b requires bytes,... not 'str'

>>> b'a %s parrot' % 'dead'.encode('ascii')   # manually encode str to bytes
b'a dead parrot'

C:\Code> py -3.3
>>> '__mod__' in dir(str), '__mod__' in dir(bytes)    # never works through 3.4     
(True, False)

C:\Code> py -3.5
>>> '__mod__' in dir(str), '__mod__' in dir(bytes)    # sometimes works in 3.5+
(True, True)

C:\Code> py -3.5
>>> 'format' in dir(str), 'format' in dir(bytes)      # but % only, not format(): why?
(True, False)

>>> 'a {} parrot'.format('dead')                      # a special-case rule is born...
'a dead parrot'

>>> b'a {} parrot'.format(b'dead')
AttributeError: 'bytes' object has no attribute 'format'

C:\Code> py -3.5 
>>> b'a %s parrot' % 'dead'                # str characters don't map to bytes
TypeError: %b requires bytes,... not 'str'

>>> b'a %s parrot' % b'dead'               # but ASCII character bytes ok here?
b'a dead parrot'

>>> ('a %s parrot'.encode('ascii') %       # it's really doing this implicitly
...               'dead'.encode('ascii'))  # but ASCII seems too narrow in 3.X 
b'a dead parrot'

>>> b'a %b parrot' % bytes([0xFF])         # ditto for binary byte values (%b=%s)
b'a \xff parrot'

>>> 'a %b parrot'.encode('ascii') % bytes([0xFF])
b'a \xff parrot'

C:\Code> py -3.5
>>> (b'a %c parrot' % 255), (b'a %c parrot' % b'\xFF')      # inserts byte values
(b'a \xff parrot', b'a \xff parrot')

>>> (b'a %d parrot' % 255), (b'a %d parrot' % b'\xFF'[0])   # inserts ASCII digits!
(b'a 255 parrot', b'a 255 parrot')

>>> (b'a %04X parrot' % 255), ('a %04X parrot' % 255).encode('ascii')   # ditto
(b'a 00FF parrot', b'a 00FF parrot')

C:\Code> py -3.5
>>> 'a %d parrot'.encode('ascii') % 255     # only an ASCII "%<code>" works!
b'a 255 parrot'

>>> 'a %d parrot'.encode('utf8') % 255      # utf8 is compatible; utf16 is not!
b'a 255 parrot'

>>> 'a %d parrot'.encode('utf16') % 255
ValueError: unsupported format character ' ' (0x0) at index 7

C:\Code> py -3.5 
>>> 'a %b parrot'.encode('latin1') % b'dead'     # ASCII-compatible text only!
b'a dead parrot'

>>> 'a %b parrot'.encode('utf16') % b'dead'
ValueError: unsupported format character ' ' (0x0) at index 7

>>> 'a %d parrot'.encode('utf16')
b'\xff\xfea\x00 \x00%\x00d\x00 \x00p\x00a\x00r\x00r\x00o\x00t\x00'

C:\Code> py -3.5 
>>> s = ('a '.encode('utf16') + b'%d' + ' parrot'.encode('utf16')) % 255
>>> s
b'\xff\xfea\x00 \x00255\xff\xfe \x00p\x00a\x00r\x00r\x00o\x00t\x00'

>>> s.decode('utf16')
UnicodeDecodeError: 'utf-16-le' codec can't decode byte 0x00 in position 24:...

>>> b'a %s parrot' % 'dead'.encode('utf16')
b'a \xff\xfed\x00e\x00a\x00d\x00 parrot'

>>> 'a %b parrot'.encode('ascii') % 'dead'.encode('utf16')
b'a \xff\xfed\x00e\x00a\x00d\x00 parrot'

C:\Code> py -3.5 
>>> b'a %s parrot' % b'dead'          # %s inserts byte values for bytes
b'a dead parrot'

>>> 'a %s parrot' % b'dead'           # but a print string for str!
"a b'dead' parrot"

>>> b'a %s parrot' % bytes([0xFF])    # ditto for non-ASCII bytes
b'a \xff parrot'

>>> 'a %s parrot' % bytes([0xFF])     # % is now a type-specific operation!
"a b'\\xff' parrot"

C:\Code> py -3 
>>> spam = 'sp\xc4\u00c4\U000000c4m'   # text formatting is for text: decoded str
>>> spam                               # original Unicode encoding is irrelevant
'spÄÄÄm'
>>> 'ham, %s, and eggs' % spam
'ham, spÄÄÄm, and eggs'

>>> code = '%s'.encode('utf16')        # format codes: decoded Unicode text
>>> code                               # ASCII requirements don't apply to str
b'\xff\xfe%\x00s\x00'
>>> ('ham, ' + code.decode('utf16') + ', and eggs') % spam
'ham, spÄÄÄm, and eggs'

>>> 'Ä %d parrot' % 255                # digits: Unicode characters (code points)
'Ä 255 parrot'                         # not ASCII-encoded text: this is 3.X!
>>> 'Ä %04X parrot' % 255
'Ä 00FF parrot'

C:\Code> py -3 
>>> s = 'Ä %d parrot \U000003A3 ᛯ \u3494' % 255 
>>> s
'Ä 255 parrot Σ ᛯ 㒔'

>>> s.encode('utf8')  
b'\xc3\x84 255 parrot \xce\xa3 \xe1\x9b\xaf \xe3\x92\x94'

C:\Code> py -3.5 
>>> b = b'the %s side of %04X'      # with the extension: ASCII implicit
>>> b % (b'bright', 255)
b'the bright side of 00FF'

>>> s = b.decode('ascii')           # without the extension: ASCII explicit
>>> s = s % ('bright', 255)         # just decode + use str % + encode
>>> s.encode('ascii')               # and this form works in all 3.X!
b'the bright side of 00FF'

C:\Code> py -3 
>>> p1 = b'bright'                  # or KISS: these work in all 3.X too!
>>> p2 = '%04X' % 255

>>> b'the ' + p1 + b' side of ' + p2.encode('ascii')
b'the bright side of 00FF'

>>> b'the $1 side of $2'.replace(b'$1', p1).replace(b'$2', p2.encode('ascii'))
b'the bright side of 00FF'

>>> b = bytearray(b'the  side of ')
>>> b[4:4] = p1
>>> b.extend(p2.encode('ascii'))
>>> b
bytearray(b'the bright side of 00FF')

[x, *iter]         # list:  unpack iter's items
(x, *iter, y)      # tuple: ditto (parenthesis or not)
{*iter, x}         # set:   ditto (values unordered and unique)
{x:y, **dict}      # dict:  unpack dict's keys/values (rightmost duplicate key wins)

C:\code> py -3.5
>>> x, y = [1, 2], (3, 4)
>>> z = [*x, 0, *y, *x]                    # unpack iterables
>>> z
[1, 2, 0, 3, 4, 1, 2]

>>> m = {'a': 1}
>>> n = {'b': 2, **m}                      # unpack dictionary
>>> n
{'a': 1, 'b': 2}

>>> n = {'b': 2, **{'b': 3}, **{'b': 4}}   # rightmost duplicate key wins
>>> n
{'b': 4}

>>> x, y = [1, 2], (3, 4)
>>> z = x + [0] + list(y) + x              # unpack iterables -- without "*"
>>> z
[1, 2, 0, 3, 4, 1, 2]

>>> m = {'a': 1}
>>> n = {'b': 2}
>>> n.update(m)                            # unpack dictionary -- without '**'
>>> n
{'a': 1, 'b': 2}

>>> n = {'b': 2}
>>> n.update({'b': 3, 'b': 4})             # ditto
>>> n
{'b': 4}

[*iter for iter in x]     # unpacking in comprehensions: abandoned in 3.5

>>> print(1, *['spam'], *[4, 'U'], '!')
1 spam 4 U !

#!/usr/bin/python3.5
import os, sys
dirname = sys.argv[1]                      # command-line arg

for name in os.listdir(dirname):           # use name strings
   path = os.path.join(dirname, name)      # type, name, path, size, modtime
   if os.path.isfile(path):
       print(name, path, os.path.getsize(path), os.path.getmtime(path))

#!/usr/bin/python3.5
import os, sys
dirname = sys.argv[1]                      # command-line arg

for dirent in os.scandir(dirname):         # use dirent objects
   if dirent.is_file():                    # type, name, path, size, modtime
       stat = dirent.stat()
       print(dirent.name, dirent.path, stat.st_size, stat.st_mtime)

/Admin-Mergeall/kingston-savagex256g/feb-2-17$ diff \
        noopt1--mergeall-date170202-time091326.txt \ 
        opt2--mergeall-date170202-time092217.txt 
0a1
> Using Python 3.5+ os.scandir() optimized variant.
4053c4054
 Phase runtime: 10.12333482701797

#!/usr/bin/python3.5
import os, sys, stat
dirname = sys.argv[1]                      # command-line arg

for name in os.listdir(dirname):           # use name strings + stat object
   path = os.path.join(dirname, name)      # type, name, path, size, modtime
   sobj = os.lstat(path)
   if stat.S_ISREG(sobj.st_mode):
       print(name, path, sobj.st_size, sobj.st_mtime)

~/Code$ py3 ls1.py /MY-STUFF/Code/mergeall > ls1.txt
~/Code$ py3 ls2.py /MY-STUFF/Code/mergeall > ls2.txt
~/Code$ py3 ls3.py /MY-STUFF/Code/mergeall > ls3.txt
~/Code$ diff ls1.txt ls2.txt
~/Code$ diff ls2.txt ls3.txt
~/Code$ cat ls1.txt
.DS_Store /MY-STUFF/Code/mergeall/.DS_Store 20484 1507832280.0
.htaccess /MY-STUFF/Code/mergeall/.htaccess 921 1507665774.0
__sloc__.py /MY-STUFF/Code/mergeall/__sloc__.py 2356 1497536861.0
backup.py /MY-STUFF/Code/mergeall/backup.py 44994 1496262548.0
...etc...

mymod.cpython-33.pyc          # from "py -3.3"
mymod.cpython-33.pyo          # from "py -3.3 -O" and "py -3.3 -OO"

mymod.cpython-35.pyc          # from "py -3.5"
mymod.cpython-35.opt-1.pyc    # from "py -3.5 -O"
mymod.cpython-35.opt-2.pyc    # from "py -3.5 -OO"

C:\Python34\python.exe            # original, through 3.4
C:\Users\yourname\AppData\Local\Programs\Python\Python35\python.exe      # 64-bit
C:\Users\yourname\AppData\Local\Programs\Python\Python35-32\python.exe   # 32-bit
C:\Program Files\Python 3.5\python.exe           # 64-bit, 32-bit on recent Windows
C:\Program Files (x86)\Python 3.5\python.exe     # 32-bit on some machines
Only Tcl/Tk 8.4 and later are supported.  Older versions are not supported. 
Use Python 3.4 or older if you cannot upgrade your Tcl/Tk libraries.

Tcl_AsyncDelete: async handler deleted by the wrong thread

>>> lens = []
>>> for line in open(r'the-crashing-folder\the-offending-file.html', encoding='utf8'):
...     lens.append(len(line))
...
>>> lens.sort()
>>> lens
[1, 1, 1, 16, 34, 41, 44, 54, 59, 79, 81, 82, 98, 100, 357, 1054, 1754, 8950, 423556]

class TextEditor:
    ...
    def onDoGrep(self, dirname, filenamepatt, grepkey, encoding):
        ...
        # start the non-GUI producer thread or process [2.2]
        spawnMode = configs.get('grepSpawnMode') or 'multiprocessing'
        grepargs = (filenamepatt, dirname, grepkey, encoding)

        if spawnMode == '_thread':
            # basic thread module (used in pymailgui with no crashes)
            myqueue = queue.Queue()
            grepargs += (myqueue,)
            _thread.start_new_thread(grepThreadProducer, grepargs)

        elif spawnMode == 'threading':
            # enhanced thread module (original coding: crashes?)
            myqueue = queue.Queue()
            grepargs += (myqueue,)
            threading.Thread(target=grepThreadProducer, args=grepargs).start()

        elif spawnMode == 'multiprocessing':
            # thread-like processes module (slower startup, faster overall?)
            myqueue = multiprocessing.Queue()
            grepargs += (myqueue,)
            multiprocessing.Process(target=grepThreadProducer, args=grepargs).start()
        else:
            assert False, 'bad grepSpawnMode setting'

        # start the GUI consumer polling loop
        self.grepThreadConsumer(grepkey, filenamepatt, encoding, myqueue, mypopup)

def spam(address: str) -> str:                 # core types
    return 'mailto:' + address

from typing import Iterable                    # new module's types
from functools import reduce

def product(vals: Iterable[int]) -> int:
    return reduce(lambda x, y: x * y, vals)

async def processrows(db):
    ...
    data = await db.fetch(querystring)            # suspend and wait
    ...

async with expr as var:           # async context managers
    suite

async for target in iter:         # async iteration loops
    suite
else:
    suite2

* pip is the preferred installer program. Starting with Python 3.4, it is
included by default with the Python binary installers.

* distutils is the original build and distribution system first added to
the Python standard library in 1998. While direct use of distutils is 
being phased out, it still laid the foundation for the current packaging
and distribution infrastructure, and it not only remains part of the 
standard library, but its name lives on in other ways (such as the name
of the mailing list used to coordinate Python packaging standards 
development).

>>> from enum import Enum
>>> class PyBooks(Enum):
        Learning5E = 2013
        Programming4E = 2011
        PocketRef5E = 2014

>>> print(PyBooks.PocketRef5E)
PyBooks.PocketRef5E
>>> PyBooks.PocketRef5E.name, PyBooks.PocketRef5E.value
('PocketRef5E', 2014)

>>> type(PyBooks.PocketRef5E)
<enum 'PyBooks'>
>>> isinstance(PyBooks.PocketRef5E, PyBooks)
True
>>> for book in PyBooks: print(book)
...
PyBooks.Learning5E
PyBooks.Programming4E
PyBooks.PocketRef5E

>>> bks = Enum('Books', 'LP5E PP4E PR5E')
>>> list(bks)
[<Books.LP5E: 1>, <Books.PP4E: 2>, <Books.PR5E: 3>]

Python
Changes
2014+

Introduction: Why This Page?

The Downside of Change

The Value of Criticism

Changes in Python 3.10 (Jan-2022)

match Python: case Bloat:

Changes in Python 3.9 (Oct-2020)

Wisdom Deprecated

Closing Words?

Changes in Python 3.8 (Oct-2019)

The Whims of the Few

Keeping It Simple

Changes in Python 3.7 (Jun-2018)

Reloads Break... Again

It's Called Bleeding for a Reason

Changes in Python 3.6 (Dec-2016)

Changes in Python 3.5 (Sep-2015)

Why Proposed Type Declarations in Python 3.5 Are a Bad Idea

The 3.X Sandbox Saga Continues: 3.5 Coroutines with "async" and "await"

Changes in Python 3.4 (Mar-2014)

For More Reading

PythonChanges2014+

Introduction: Why This Page?

The Downside of Change

The Value of Criticism

Changes in Python 3.10 (Jan-2022)

match Python: case Bloat:

Changes in Python 3.9 (Oct-2020)

Wisdom Deprecated

Closing Words?

Changes in Python 3.8 (Oct-2019)

The Whims of the Few

Keeping It Simple

Changes in Python 3.7 (Jun-2018)

Reloads Break... Again

It's Called Bleeding for a Reason

Changes in Python 3.6 (Dec-2016)

Changes in Python 3.5 (Sep-2015)

Why Proposed Type Declarations in Python 3.5 Are a Bad Idea

The 3.X Sandbox Saga Continues: 3.5 Coroutines with "async" and "await"

Changes in Python 3.4 (Mar-2014)

For More Reading

Python
Changes
2014+