Scrapy导出欠套型JSON

目录

Scrapy导出欠套型JSON

scrapy如何导出类型如下结构的JSON：

[
{
	"pingPai": ["ALPINA"],
	"carTypes": [{
		"carType": ["ALPINA"],
		"carNames": {
			"carName": ["ALPINA B4",
			"ALPINA B3",
			"ALPINA D5",
			"ALPINA B7",
			"ALPINA XD3"]
		}
	}],
	"picUrl": "https://car3.autoimg.cn/cardfs/series/g27/M05/AB/2E/100x100_f40_autohomecar__wKgHHls8hiKADrqGAABK67H4HUI503.png"
},
{
	"pingPai": ["ABT"],
	"carTypes": [{
		"carType": ["ABT"],
		"carNames": {
			"carName": ["ABT A3",
			"ABT A5",
			"ABT TT"]
		}
	}],
	"picUrl": "https://car3.autoimg.cn/cardfs/series/g30/M07/B0/47/100x100_f40_autohomecar__wKgHPls9vLOAHILAAAAWGGhA_W0282.png"
}    
]

解决核心

Scrapy导出欠套型JSON实质是对列表的操作

准备知识

  • python合并多个列表，直接用“+”
list1 = [1,2,3]
list2 = [5,6,7]
print(list1+list2)
# 输出：
[1, 2, 3, 5, 6, 7]
  • python合并多个字典，用“update()”(也可以用其它方法，这里只讲update)（在此文中实际上没有用到）
dic1 = {'a':'1','b':'2'}
dic2 = {'c':'3','d':'4'}
dic1.update(dic2)
print(dic1)
# 输出：
{'a': '1', 'b': '2', 'c': '3', 'd': '4'}
  • xpath匹配得到的结果实际上是一个列表
    比如，xpath匹配到一行数据就是“[X]”，X是所匹配的值
    xpath匹配到多行数据就是“[X,Y,Z....]”

解决方法

观察如上欠套JSON，1级节点是：“pingPai”、“carTypes”、“picUrl”三个字段，根据scrapy定义items.py文件的特性，我们只需要定义这三个一级节点，定义为：
打开items.py文件，添加如下代码：

class CarModelItem(scrapy.Item):
    pingPai = scrapy.Field()  # 品牌
    carTypes = scrapy.Field()  # 车型
    picUrl = scrapy.Field()  # 品牌图片

注意

此处也可以不用定义 items.py文件 直接在导出的pipelines.py文件里面使用json.dumps系列化（把对象转换为字节序列的过程称为对象的序列化。把字节序列恢复为对象的过程称为对象的反序列化。）（json.dumps将一个Python数据结构转换为JSON，json.loads将一个JSON编码的字符串转换回一个Python数据结构）原理都是一样的，只是运用了系列化而已，本文不作讨论。

要生成欠套型的JSON，我们只需要在carTypes列表内再添加列表就行（添加值为字典“{}”类型的列表就可以了）。
比如我们要爬取地址为“https://www.autohome.com.cn/grade/carhtml/A.html”这个地址的内容，打开浏览器查看效果如下：

查看源代码，图片和1级节点在“dl/dt”内，如下图：

第二节点和第三节点在“dl/dd”内，如下图：

这里是比较难处理的地方，一般这里我们要定义的列表为：列表内再添加列表（值为字典）的数据格式才能满足需求。
直接上代码：

class GetcarmodelSpider(scrapy.Spider):
    name = "GetCarModel"
    allowed_domains = ["www.autohome.com.cn"]
    chars = [
        "A",
        """
        "B",
        "C",
        "D",
        "F",
        "G",
        "H",
        "J",
        "K",
        "L",
        "M",
        "N",
        "O",
        "P",
        "Q",
        "R",
        "S",
        "T",
        "W",
        "X",
        "Y",
        "Z",""",
    ]
    start_urls = [
        "https://www.autohome.com.cn/grade/carhtml/%s.html" % i2 for i2 in chars
    ]

    def parse(self, response):
        dtArray = response.xpath("//dl[@id]")
        for dt in dtArray:
            pingPai = dt.xpath("./dt/div/a/text()").extract()
            pingPaiPicArr = dt.xpath("./dt/a/img/@src").extract()
            pingPaiPic = ""
            # 这里图片其实只有一张图片
            for cti in pingPaiPicArr:
                # carTypeImg = "http://" + cti[2:]
                pingPaiPic = parse.urljoin(response.url, cti)

            carTypesTemp = dt.xpath("./dd/div[@class='h3-tit']")
            carTypes = []
            for pp in carTypesTemp:
                print(">>>>>>>>>>>>>>>>>>>>>>", pp.xpath("./a/text()").extract())
                carTypes += [
                    {"carType": pp.xpath("./a/text()").extract(), "carNames": {}}
                ]

            # 获取具体名称
            carNameArray = dt.xpath("./dd/ul[@class='rank-list-ul']")
            carNames = []
            for cn in carNameArray:
                # 直接定义值为字典类型的列表，这样在循环第X次的时候取值就是carNames[X]
                carNames += [{"carName": cn.xpath("./li/h4/a/text()").extract()}]
                print(".......", [{"carName": cn.xpath("./li/h4/a/text()").extract()}])

            for i in range(len(carTypes)):
                try:
                    carTypes[i]["carNames"] = carNames[i]
                except Exception as e:
                    print(e)


            print("pingPai:", pingPai)
            print("pingPaiPic:", pingPaiPic)
            print("carTypes:", carTypes)
            print("carNames:", carNames)

            carModel = CarModelItem()
            carModel["pingPai"] = pingPai
            carModel["carTypes"] = carTypes
            carModel["picUrl"] = pingPaiPic
            yield carModel

注意，导出JSON方法这里不再说明，自行搜索，网上一大堆
运行代码，得到导出的JSON文件如下：

[{
	"pingPai": ["奥迪"],
	"carTypes": [{
		"carType": ["一汽-大众奥迪"],
		"carNames": {
			"carName": ["奥迪Q2L新能源",
			"奥迪A3",
			"奥迪A4L",
			"奥迪A6L",
			"奥迪Q2L",
			"奥迪Q3",
			"奥迪Q5L",
			"奥迪A6L新能源",
			"奥迪Q4",
			"奥迪A4",
			"奥迪A6",
			"奥迪Q5"]
		}
	},
	{
		"carType": ["Audi Sport"],
		"carNames": {
			"carName": ["奥迪RS 3",
			"奥迪RS 4",
			"奥迪RS 5",
			"奥迪RS 6",
			"奥迪RS 7",
			"奥迪R8",
			"奥迪TT RS",
			"奥迪RS Q3",
			"奥迪RSQ e-tron"]
		}
	},
	{
		"carType": ["奥迪(进口)"],
		"carNames": {
			"carName": ["奥迪e-tron",
			"奥迪A3(进口)",
			"奥迪S3",
			"奥迪A4(进口)",
			"奥迪A5",
			"奥迪S4",
			"奥迪S5",
			"奥迪A6(进口)",
			"奥迪S6",
			"奥迪A7",
			"奥迪S7",
			"奥迪A8",
			"奥迪Q7",
			"奥迪Q7新能源",
			"奥迪TT",
			"奥迪TTS",
			"奥迪A0",
			"奥迪A1",
			"奥迪S1",
			"e-tron Concept",
			"奥迪AI:ME",
			"奥迪A6新能源(进口)",
			"奥迪A7新能源",
			"奥迪Aicon",
			"奥迪e-tron GT",
			"Prologue",
			"奥迪A8新能源",
			"奥迪A9",
			"奥迪S8",
			"allroad",
			"奥迪Q2",
			"奥迪SQ2",
			"奥迪Q3(进口)",
			"奥迪Q4(进口)",
			"奥迪Q4新能源(进口)",
			"奥迪TT offroad",
			"h-tron quattro",
			"奥迪Elaine",
			"奥迪Q5(进口)",
			"奥迪Q5新能源(进口)",
			"奥迪SQ5",
			"奥迪Q8",
			"奥迪SQ7",
			"奥迪Q9",
			"e-tron Vision Gran Turismo",
			"quattro",
			"奥迪PB18",
			"奥迪R18",
			"奥迪Urban",
			"奥迪A2",
			"奥迪80",
			"奥迪A3新能源(进口)",
			"奥迪Coupe",
			"奥迪100",
			"Crosslane Coupe",
			"奥迪Cross",
			"Nanuk"]
		}
	}],
	"picUrl": "https://car2.autoimg.cn/cardfs/series/g26/M0B/AE/B3/100x100_f40_autohomecar__wKgHEVs9u5WAV441AAAKdxZGE4U148.png"
},
{
	"pingPai": ["阿斯顿·马丁"],
	"carTypes": [{
		"carType": ["阿斯顿·马丁"],
		"carNames": {
			"carName": ["Rapide",
			"V8 Vantage",
			"Vanquish",
			"阿斯顿·马丁DB11",
			"阿斯顿·马丁DBS",
			"Cygnet",
			"Rapide E",
			"阿斯顿·马丁DBX",
			"V12 Vantage",
			"阿斯顿·马丁DB9",
			"AM-RB 003",
			"Heritage EV",
			"Virage",
			"Vulcan",
			"阿斯顿·马丁CC100",
			"阿斯顿·马丁DB10",
			"阿斯顿·马丁DB5",
			"阿斯顿·马丁DP-100",
			"战神",
			"拉共达Taraf",
			"Ulster",
			"V12 Zagato",
			"阿斯顿·马丁DB6",
			"阿斯顿·马丁One-77"]
		}
	}],
	"picUrl": "https://car2.autoimg.cn/cardfs/series/g26/M06/AE/B5/100x100_f40_autohomecar__wKgHEVs9u6GAPWN8AAAYsmBsCWs847.png"
},
{
	"pingPai": ["AC Schnitzer"],
	"carTypes": [{
		"carType": ["AC Schnitzer"],
		"carNames": {
			"carName": ["AC Schnitzer 3系",
			"AC Schnitzer M4",
			"AC Schnitzer 7系",
			"AC Schnitzer X6",
			"AC Schnitzer X5"]
		}
	}],
	"picUrl": "https://car3.autoimg.cn/cardfs/series/g27/M01/B0/62/100x100_f40_autohomecar__ChcCQFs9vBKAO3YSAAAW0WOWvRc555.png"
},
{
	"pingPai": ["安凯客车"],
	"carTypes": [{
		"carType": ["安凯客车"],
		"carNames": {
			"carName": ["宝斯通"]
		}
	}],
	"picUrl": "https://car2.autoimg.cn/cardfs/series/g29/M00/AB/C8/100x100_f40_autohomecar__ChcCSFs8riCAYVA2AAApQLgf8a0969.png"
},
{
	"pingPai": ["阿尔法·罗密欧"],
	"carTypes": [{
		"carType": ["阿尔法·罗密欧"],
		"carNames": {
			"carName": ["Giulia",
			"Stelvio",
			"MiTo",
			"Giulietta",
			"Tonale",
			"ALFA 4C",
			"Disco Volante",
			"Gloria",
			"ALFA 147",
			"ALFA 156",
			"ALFA 159",
			"ALFA 166",
			"ALFA 2uettottanta",
			"ALFA 8C",
			"ALFA GT",
			"ALFA S.Z.",
			"ALFA TZ3"]
		}
	}],
	"picUrl": "https://car2.autoimg.cn/cardfs/series/g26/M05/B0/29/100x100_f40_autohomecar__ChcCP1s9u5qAemANAABON_GMdvI451.png"
},
{
	"pingPai": ["ALPINA"],
	"carTypes": [{
		"carType": ["ALPINA"],
		"carNames": {
			"carName": ["ALPINA B4",
			"ALPINA B3",
			"ALPINA D5",
			"ALPINA B7",
			"ALPINA XD3"]
		}
	}],
	"picUrl": "https://car3.autoimg.cn/cardfs/series/g27/M05/AB/2E/100x100_f40_autohomecar__wKgHHls8hiKADrqGAABK67H4HUI503.png"
},
{
	"pingPai": ["ABT"],
	"carTypes": [{
		"carType": ["ABT"],
		"carNames": {
			"carName": ["ABT A3",
			"ABT A5",
			"ABT TT"]
		}
	}],
	"picUrl": "https://car3.autoimg.cn/cardfs/series/g30/M07/B0/47/100x100_f40_autohomecar__wKgHPls9vLOAHILAAAAWGGhA_W0282.png"
},
{
	"pingPai": ["AEV ROBOTICS"],
	"carTypes": [{
		"carType": ["AEV ROBOTICS"],
		"carNames": {
			"carName": ["Modular Vehicle System"]
		}
	}],
	"picUrl": "https://car2.autoimg.cn/cardfs/series/g3/M02/58/D3/autohomecar__ChcCRVw0TJaAM8BmAAAS-7AD7DQ372.png"
},
{
	"pingPai": ["Agile Automotive"],
	"carTypes": [{
		"carType": ["Agile Automotive"],
		"carNames": {
			"carName": ["Agile Automotive SC122",
			"Agile Automotive SCX"]
		}
	}],
	"picUrl": "https://car2.autoimg.cn/cardfs/series/g26/M09/AF/8C/100x100_f40_autohomecar__wKgHHVs9r62AIbiYAAAvAsqdpoA594.png"
},
{
	"pingPai": ["Apollo"],
	"carTypes": [{
		"carType": ["Apollo"],
		"carNames": {
			"carName": ["Apollo N",
			"Arrow",
			"Intensa Emozione"]
		}
	}],
	"picUrl": "https://car3.autoimg.cn/cardfs/series/g28/M06/B0/C6/100x100_f40_autohomecar__ChcCR1s90RGASBRgAACz67wh_68723.png"
},
{
	"pingPai": ["Arash"],
	"carTypes": [{
		"carType": ["Arash"],
		"carNames": {
			"carName": ["AF8 Cassini",
			"Arash AF10"]
		}
	}],
	"picUrl": "https://car3.autoimg.cn/cardfs/series/g30/M05/AA/D4/100x100_f40_autohomecar__wKgHHFs8n1CAVhcNAAAV3xEAiDM531.png"
},
{
	"pingPai": ["ARCFOX"],
	"carTypes": [{
		"carType": ["北汽新能源"],
		"carNames": {
			"carName": ["ARCFOX-1",
			"ARCFOX ECF Concept",
			"ARCFOX-7",
			"ARCFOX-GT"]
		}
	}],
	"picUrl": "https://car3.autoimg.cn/cardfs/series/g27/M02/AB/F7/100x100_f40_autohomecar__ChcCQFs8nA6AP-h5AABsvxhHw3E709.png"
},
{
	"pingPai": ["Aria"],
	"carTypes": [{
		"carType": ["Aria"],
		"carNames": {
			"carName": ["Aria FXE"]
		}
	}],
	"picUrl": "https://car3.autoimg.cn/cardfs/series/g28/M0B/B0/0D/100x100_f40_autohomecar__wKgHI1s9r2iAJwIXAAAIBShzq60456.png"
},
{
	"pingPai": ["ATS"],
	"carTypes": [{
		"carType": ["ATS"],
		"carNames": {
			"carName": ["ATS GT"]
		}
	}],
	"picUrl": "https://car2.autoimg.cn/cardfs/series/g26/M08/D7/D3/autohomecar__ChsEe1wYwKmAY2p9AAA1NP0jCHk594.png"
},
{
	"pingPai": ["Aurus"],
	"carTypes": [{
		"carType": ["Aurus"],
		"carNames": {
			"carName": ["Senat"]
		}
	}],
	"picUrl": "https://car2.autoimg.cn/cardfs/series/g27/M07/F3/E1/autohomecar__ChcCQFuN6WiAcztKAAAsLfBmU9g074.png"
},
{
	"pingPai": ["艾康尼克"],
	"carTypes": [{
		"carType": ["艾康尼克ICONIQ Motors"],
		"carNames": {
			"carName": ["MUSE",
			"艾康尼克七系"]
		}
	}],
	"picUrl": "https://car2.autoimg.cn/cardfs/series/g29/M0A/A9/EC/100x100_f40_autohomecar__wKgHG1s8iP6ASbjTAAAOIwskkzo314.png"
},
{
	"pingPai": ["爱驰"],
	"carTypes": [{
		"carType": ["爱驰汽车"],
		"carNames": {
			"carName": ["爱驰U5",
			"爱驰U7",
			"RG Nathalie"]
		}
	}],
	"picUrl": "https://car3.autoimg.cn/cardfs/series/g29/M09/A9/9B/100x100_f40_autohomecar__wKgHG1s8fwqAOp3IAAALEeTkn6c536.png"
}]
【推广】 免费学中医，健康全家人
原文地址：https://www.cnblogs.com/zh672903/p/11018891.html