python GIL 和线程锁和进程锁

Are locks unnecessary in multi-threaded Python code because of the GIL?

https://stackoverflow.com/questions/105095/are-locks-unnecessary-in-multi-threaded-python-code-because-of-the-gil

由于GIL的存在，导致所有 python的多线程只能是并发，并不是并行。

但是这样的情况下，共享变量，同一时间只能被一个线程访问，那么为什么多线程模块还设计 lock？

If you are relying on an implementation of Python that has a Global Interpreter Lock (i.e. CPython) and writing multithreaded code, do you really need locks at all?

If the GIL doesn't allow multiple instructions to be executed in parallel, wouldn't shared data be unnecessary to protect?

sorry if this is a dumb question, but it is something I have always wondered about Python on multi-processor/core machines.

same thing would apply to any other language implementation that has a GIL.

answer -- data inconsistent

对于进程间的共享变量，多个线程访问，没有物理上的冲突。

但是对于业务上的逻辑，会导致业务上的冲突，读取后，然后在更新，两个进程交替执行，会导致数据不一致性。

You will still need locks if you share state between threads. The GIL only protects the interpreter internally. You can still have inconsistent updates in your own code.

Here, your code can be interrupted between reading the shared state (balance = shared_balance) and writing the changed result back (shared_balance = balance), causing a lost update. The result is a random value for the shared state.

To make the updates consistent, run methods would need to lock the shared state around the read-modify-write sections (inside the loops) or have some way to detect when the shared state had changed since it was read.

#!/usr/bin/env python
import threading

shared_balance = 0

class Deposit(threading.Thread):
    def run(self):
        for _ in xrange(1000000):
            global shared_balance
            balance = shared_balance
            balance += 100
            shared_balance = balance

class Withdraw(threading.Thread):
    def run(self):
        for _ in xrange(1000000):
            global shared_balance
            balance = shared_balance
            balance -= 100
            shared_balance = balance

threads = [Deposit(), Withdraw()]

for thread in threads:
    thread.start()

for thread in threads:
    thread.join()

print shared_balance

solution 1 - thread lock

https://docs.python.org/3.6/library/threading.html#lock-objects

用法

https://haicoder.net/python/python-thread-lock.html

import threading
num = 0
# 创建互斥锁
lock = threading.Lock()
def handler_incry():
    global num
    for i in range(100000):
        num += 1
    print("handler_incry done, num =", num)
def handler_decry():
    global num
    for i in range(100000):
        num -= 1
    print("handler_decry done, num =", num)
if __name__ == '__main__':
    print("嗨客网(www.haicoder.net)")
    # 创建线程
    t1 = threading.Thread(target=handler_incry)
    t2 = threading.Thread(target=handler_decry)
    # 启动线程
    t1.start()
    t2.start()
    t1.join()
    t2.join()

改造

https://github.com/fanqingsong/code_snippet/blob/master/python/multithread/threadlock_plock.py

#!/usr/bin/env python
import threading
from multiprocessing import Process, Lock

shared_balance = 0

lock = Lock()

class Deposit(threading.Thread):
    def run(self):
        for _ in xrange(1000000):
            lock.acquire()
            global shared_balance
            balance = shared_balance
            balance += 100
            shared_balance = balance
            lock.release()

class Withdraw(threading.Thread):
    def run(self):
        for _ in xrange(1000000):
            lock.acquire()
            global shared_balance
            balance = shared_balance
            balance -= 100
            shared_balance = balance
            lock.release()

threads = [Deposit(), Withdraw()]

for thread in threads:
    thread.start()

for thread in threads:
    thread.join()

print shared_balance

solution2 -- process lock

https://docs.python.org/3.6/library/multiprocessing.html#multiprocessing.Lock

A non-recursive lock object: a close analog of threading.Lock. Once a process or thread has acquired a lock, subsequent attempts to acquire it from any process or thread will block until it is released; any process or thread may release it. The concepts and behaviors of threading.Lock as it applies to threads are replicated here in multiprocessing.Lock as it applies to either processes or threads, except as noted.

改造

https://github.com/fanqingsong/code_snippet/blob/master/python/multithread/threadlock_plock.py

使用进程锁也可以实现关键代码互斥。

进程级别的互斥，可以兼容线程级的互斥。

#!/usr/bin/env python
import threading
from multiprocessing import Process, Lock

shared_balance = 0

lock = Lock()

class Deposit(threading.Thread):
    def run(self):
        for _ in xrange(1000000):
            lock.acquire()
            global shared_balance
            balance = shared_balance
            balance += 100
            shared_balance = balance
            lock.release()

class Withdraw(threading.Thread):
    def run(self):
        for _ in xrange(1000000):
            lock.acquire()
            global shared_balance
            balance = shared_balance
            balance -= 100
            shared_balance = balance
            lock.release()

threads = [Deposit(), Withdraw()]

for thread in threads:
    thread.start()

for thread in threads:
    thread.join()

print shared_balance

Shared memory + proccess lock == cover thread and process data sharing

Shared memory

https://docs.python.org/3.6/library/multiprocessing.html#sharing-state-between-processes

Data can be stored in a shared memory map using Value or Array. For example, the following code

from multiprocessing import Process, Value, Array

def f(n, a):
    n.value = 3.1415927
    for i in range(len(a)):
        a[i] = -a[i]

if __name__ == '__main__':
    num = Value('d', 0.0)
    arr = Array('i', range(10))

    p = Process(target=f, args=(num, arr))
    p.start()
    p.join()

    print(num.value)
    print(arr[:])

Integration of Both