Preston_PLB
/
kernel-aes67
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

This update includes the following changes: 

API:
 
 - Avoid unnecessary copying in scomp for trivial SG lists.
 
 Algorithms:
 
 - Optimise NEON CCM implementation on ARM64.
 
 Drivers:
 
 - Add queue stop/query debugfs support in hisilicon/qm.
 -
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEEn51F/lCuNhUwmDeSxycdCkmxi6cFAmXxDbcACgkQxycdCkmx
 i6dCzw//dI5CuszRMalZu8fwvh7K48d4GoY/5uuc1vPK6yLyc5dl6CzPsrKhmBGO
 Ip2fUCFoO+ViWl5tw8SjOOEnPRztETrIGlOM5PHOIxRuSA6lbcsdo+EQja4WBDo1
 fGUNaNVW9eY9XsaUw46uwxaicd+xOqRGJURKB2XuzSHACln/QrNy74aVdbc6VhaK
 Jnh4o2blsEh7jcVAK2ahNmmDm739w8C462Go4UIl8xcM693HtjUUFx7TALpI0iC+
 BWxctAV9IzA/siUwztvwwlo+V1KZbjZFD5XfC/erkV0gFs7ll8IcuDFnZ1suMvt2
 Z+6OVhghkkvMJJEl1qKNxVJITOUdXH0yKWqTbEHuFlV7VV+9hjUlWlvRVm7ZIrXf
 ZXi/OpBJRAIBB6fJqy1RAvfIZUR8Dl8i8RKLVawdqFLbB+XCOiIK7a8+5hCKaWsU
 0DVFNCfiJFPIiByzxXV+4Jt/hzxx/qseIbTiUShdoXeHOVWMgH5H255MWJv5qjVy
 aSwgQLX6MlY9Pj+w5cPHAUjOiGjiAa3iKwsbnmo5lszsPl4KR407gaJnV7VI4s6R
 fhqAgIjJ4Ik2lHzQRM89QU2OOogVBPs+FHEkk98lak5CATpjUq97iLbTharAg3dc
 l8FLoSB9+NM+RCbnzUXuOIvoU/okSM4+j6a3xAnLQH6OxBOtk3Y=
 =Uw2R
 -----END PGP SIGNATURE-----

Merge tag 'v6.9-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6

Pull crypto updates from Herbert Xu:
 "API:

   - Avoid unnecessary copying in scomp for trivial SG lists

  Algorithms:

   - Optimise NEON CCM implementation on ARM64

  Drivers:

   - Add queue stop/query debugfs support in hisilicon/qm

   - Intel qat updates and cleanups"

* tag 'v6.9-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6: (79 commits)
  Revert "crypto: remove CONFIG_CRYPTO_STATS"
  crypto: scomp - remove memcpy if sg_nents is 1 and pages are lowmem
  crypto: tcrypt - add ffdhe2048(dh) test
  crypto: iaa - fix the missing CRYPTO_ALG_ASYNC in cra_flags
  crypto: hisilicon/zip - fix the missing CRYPTO_ALG_ASYNC in cra_flags
  hwrng: hisi - use dev_err_probe
  MAINTAINERS: Remove T Ambarus from few mchp entries
  crypto: iaa - Fix comp/decomp delay statistics
  crypto: iaa - Fix async_disable descriptor leak
  dt-bindings: rng: atmel,at91-trng: add sam9x7 TRNG
  dt-bindings: crypto: add sam9x7 in Atmel TDES
  dt-bindings: crypto: add sam9x7 in Atmel SHA
  dt-bindings: crypto: add sam9x7 in Atmel AES
  crypto: remove CONFIG_CRYPTO_STATS
  crypto: dh - Make public key test FIPS-only
  crypto: rockchip - fix to check return value
  crypto: jitter - fix CRYPTO_JITTERENTROPY help text
  crypto: qat - make ring to service map common for QAT GEN4
  crypto: qat - fix ring to service map for dcc in 420xx
  crypto: qat - fix ring to service map for dcc in 4xxx
  ...
This commit is contained in:
Linus Torvalds 2024-03-15 14:46:54 -07:00
parent 4138f02288 6a8dbd71a7
commit c8e7699616
100 changed files with 1450 additions and 1016 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
Documentation/ABI/testing/debugfs-driver-qat
View File

@ -81,3 +81,29 @@ Description: (RO) Read returns, for each Acceleration Engine (AE), the number
<N>: Number of Compress and Verify (CnV) errors and type
of the last CnV error detected by Acceleration
Engine N.
What: /sys/kernel/debug/qat_<device>_<BDF>/heartbeat/inject_error
Date: March 2024
KernelVersion: 6.8
Contact: qat-linux@intel.com
Description: (WO) Write to inject an error that simulates an heartbeat
failure. This is to be used for testing purposes.
After writing this file, the driver stops arbitration on a
random engine and disables the fetching of heartbeat counters.
If a workload is running on the device, a job submitted to the
accelerator might not get a response and a read of the
`heartbeat/status` attribute might report -1, i.e. device
unresponsive.
The error is unrecoverable thus the device must be restarted to
restore its functionality.
This attribute is available only when the kernel is built with
CONFIG_CRYPTO_DEV_QAT_ERROR_INJECTION=y.
A write of 1 enables error injection.
The following example shows how to enable error injection::
# cd /sys/kernel/debug/qat_<device>_<BDF>
# echo 1 > heartbeat/inject_error

22
Documentation/ABI/testing/debugfs-hisi-hpre
View File

@ -111,6 +111,28 @@ Description: QM debug registers(regs) read hardware register value. This
node is used to show the change of the qm register values. This
node can be help users to check the change of register values.
What: /sys/kernel/debug/hisi_hpre/<bdf>/qm/qm_state
Date: Jan 2024
Contact: linux-crypto@vger.kernel.org
Description: Dump the state of the device.
0: busy, 1: idle.
Only available for PF, and take no other effect on HPRE.
What: /sys/kernel/debug/hisi_hpre/<bdf>/qm/dev_timeout
Date: Feb 2024
Contact: linux-crypto@vger.kernel.org
Description: Set the wait time when stop queue fails. Available for both PF
and VF, and take no other effect on HPRE.
0: not wait(default), others value: wait dev_timeout * 20 microsecond.
What: /sys/kernel/debug/hisi_hpre/<bdf>/qm/dev_state
Date: Feb 2024
Contact: linux-crypto@vger.kernel.org
Description: Dump the stop queue status of the QM. The default value is 0,
if dev_timeout is set, when stop queue fails, the dev_state
will return non-zero value. Available for both PF and VF,
and take no other effect on HPRE.
What: /sys/kernel/debug/hisi_hpre/<bdf>/hpre_dfx/diff_regs
Date: Mar 2022
Contact: linux-crypto@vger.kernel.org

22
Documentation/ABI/testing/debugfs-hisi-sec
View File

@ -91,6 +91,28 @@ Description: QM debug registers(regs) read hardware register value. This
node is used to show the change of the qm register values. This
node can be help users to check the change of register values.
What: /sys/kernel/debug/hisi_sec2/<bdf>/qm/qm_state
Date: Jan 2024
Contact: linux-crypto@vger.kernel.org
Description: Dump the state of the device.
0: busy, 1: idle.
Only available for PF, and take no other effect on SEC.
What: /sys/kernel/debug/hisi_sec2/<bdf>/qm/dev_timeout
Date: Feb 2024
Contact: linux-crypto@vger.kernel.org
Description: Set the wait time when stop queue fails. Available for both PF
and VF, and take no other effect on SEC.
0: not wait(default), others value: wait dev_timeout * 20 microsecond.
What: /sys/kernel/debug/hisi_sec2/<bdf>/qm/dev_state
Date: Feb 2024
Contact: linux-crypto@vger.kernel.org
Description: Dump the stop queue status of the QM. The default value is 0,
if dev_timeout is set, when stop queue fails, the dev_state
will return non-zero value. Available for both PF and VF,
and take no other effect on SEC.
What: /sys/kernel/debug/hisi_sec2/<bdf>/sec_dfx/diff_regs
Date: Mar 2022
Contact: linux-crypto@vger.kernel.org

22
Documentation/ABI/testing/debugfs-hisi-zip
View File

@ -104,6 +104,28 @@ Description: QM debug registers(regs) read hardware register value. This
node is used to show the change of the qm registers value. This
node can be help users to check the change of register values.
What: /sys/kernel/debug/hisi_zip/<bdf>/qm/qm_state
Date: Jan 2024
Contact: linux-crypto@vger.kernel.org
Description: Dump the state of the device.
0: busy, 1: idle.
Only available for PF, and take no other effect on ZIP.
What: /sys/kernel/debug/hisi_zip/<bdf>/qm/dev_timeout
Date: Feb 2024
Contact: linux-crypto@vger.kernel.org
Description: Set the wait time when stop queue fails. Available for both PF
and VF, and take no other effect on ZIP.
0: not wait(default), others value: wait dev_timeout * 20 microsecond.
What: /sys/kernel/debug/hisi_zip/<bdf>/qm/dev_state
Date: Feb 2024
Contact: linux-crypto@vger.kernel.org
Description: Dump the stop queue status of the QM. The default value is 0,
if dev_timeout is set, when stop queue fails, the dev_state
will return non-zero value. Available for both PF and VF,
and take no other effect on ZIP.
What: /sys/kernel/debug/hisi_zip/<bdf>/zip_dfx/diff_regs
Date: Mar 2022
Contact: linux-crypto@vger.kernel.org

20
Documentation/ABI/testing/sysfs-driver-qat
View File

@ -141,3 +141,23 @@ Description:
64
This attribute is only available for qat_4xxx devices.
What: /sys/bus/pci/devices/<BDF>/qat/auto_reset
Date: March 2024
KernelVersion: 6.8
Contact: qat-linux@intel.com
Description: (RW) Reports the current state of the autoreset feature
for a QAT device
Write to the attribute to enable or disable device auto reset.
Device auto reset is disabled by default.
The values are:
* 1/Yy/on: auto reset enabled. If the device encounters an
unrecoverable error, it will be reset automatically.
* 0/Nn/off: auto reset disabled. If the device encounters an
unrecoverable error, it will not be reset.
This attribute is only available for qat_4xxx devices.

6
Documentation/devicetree/bindings/crypto/atmel,at91sam9g46-aes.yaml
View File

@ -12,7 +12,11 @@ maintainers:
properties:
compatible:
const: atmel,at91sam9g46-aes
oneOf:
- const: atmel,at91sam9g46-aes
- items:
- const: microchip,sam9x7-aes
- const: atmel,at91sam9g46-aes
reg:
maxItems: 1

6
Documentation/devicetree/bindings/crypto/atmel,at91sam9g46-sha.yaml
View File

@ -12,7 +12,11 @@ maintainers:
properties:
compatible:
const: atmel,at91sam9g46-sha
oneOf:
- const: atmel,at91sam9g46-sha
- items:
- const: microchip,sam9x7-sha
- const: atmel,at91sam9g46-sha
reg:
maxItems: 1

6
Documentation/devicetree/bindings/crypto/atmel,at91sam9g46-tdes.yaml
View File

@ -12,7 +12,11 @@ maintainers:
properties:
compatible:
const: atmel,at91sam9g46-tdes
oneOf:
- const: atmel,at91sam9g46-tdes
- items:
- const: microchip,sam9x7-tdes
- const: atmel,at91sam9g46-tdes
reg:
maxItems: 1

1
Documentation/devicetree/bindings/crypto/qcom,inline-crypto-engine.yaml
View File

@ -14,6 +14,7 @@ properties:
items:
- enum:
- qcom,sa8775p-inline-crypto-engine
- qcom,sc7180-inline-crypto-engine
- qcom,sm8450-inline-crypto-engine
- qcom,sm8550-inline-crypto-engine
- qcom,sm8650-inline-crypto-engine

1
Documentation/devicetree/bindings/crypto/qcom-qce.yaml
View File

@ -45,6 +45,7 @@ properties:
- items:
- enum:
- qcom,sc7280-qce
- qcom,sm6350-qce
- qcom,sm8250-qce
- qcom,sm8350-qce
- qcom,sm8450-qce

4
Documentation/devicetree/bindings/rng/atmel,at91-trng.yaml
View File

@ -21,6 +21,10 @@ properties:
- enum:
- microchip,sama7g5-trng
- const: atmel,at91sam9g45-trng
- items:
- enum:
- microchip,sam9x7-trng
- const: microchip,sam9x60-trng
clocks:
maxItems: 1

25
MAINTAINERS
View File

@ -10377,12 +10377,17 @@ M: Nayna Jain <nayna@linux.ibm.com>
M: Paulo Flabiano Smorigo <pfsmorigo@gmail.com>
L: linux-crypto@vger.kernel.org
S: Supported
F: drivers/crypto/vmx/Kconfig
F: drivers/crypto/vmx/Makefile
F: drivers/crypto/vmx/aes*
F: drivers/crypto/vmx/ghash*
F: drivers/crypto/vmx/ppc-xlate.pl
F: drivers/crypto/vmx/vmx.c
F: arch/powerpc/crypto/Kconfig
F: arch/powerpc/crypto/Makefile
F: arch/powerpc/crypto/aes.c
F: arch/powerpc/crypto/aes_cbc.c
F: arch/powerpc/crypto/aes_ctr.c
F: arch/powerpc/crypto/aes_xts.c
F: arch/powerpc/crypto/aesp8-ppc.*
F: arch/powerpc/crypto/ghash.c
F: arch/powerpc/crypto/ghashp8-ppc.pl
F: arch/powerpc/crypto/ppc-xlate.pl
F: arch/powerpc/crypto/vmx.c
IBM ServeRAID RAID DRIVER
S: Orphan
@ -12456,7 +12461,6 @@ F: drivers/*/*/*pasemi*
F: drivers/*/*pasemi*
F: drivers/char/tpm/tpm_ibmvtpm*
F: drivers/crypto/nx/
F: drivers/crypto/vmx/
F: drivers/i2c/busses/i2c-opal.c
F: drivers/net/ethernet/ibm/ibmveth.*
F: drivers/net/ethernet/ibm/ibmvnic.*
@ -14304,7 +14308,6 @@ F: drivers/misc/xilinx_tmr_manager.c
MICROCHIP AT91 DMA DRIVERS
M: Ludovic Desroches <ludovic.desroches@microchip.com>
M: Tudor Ambarus <tudor.ambarus@linaro.org>
L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
L: dmaengine@vger.kernel.org
S: Supported
@ -14353,9 +14356,8 @@ F: Documentation/devicetree/bindings/media/microchip,csi2dc.yaml
F: drivers/media/platform/microchip/microchip-csi2dc.c
MICROCHIP ECC DRIVER
M: Tudor Ambarus <tudor.ambarus@linaro.org>
L: linux-crypto@vger.kernel.org
S: Maintained
S: Orphan
F: drivers/crypto/atmel-ecc.*
MICROCHIP EIC DRIVER
@ -14460,9 +14462,8 @@ S: Maintained
F: drivers/mmc/host/atmel-mci.c
MICROCHIP NAND DRIVER
M: Tudor Ambarus <tudor.ambarus@linaro.org>
L: linux-mtd@lists.infradead.org
S: Supported
S: Orphan
F: Documentation/devicetree/bindings/mtd/atmel-nand.txt
F: drivers/mtd/nand/raw/atmel/*

13
arch/arm/crypto/sha256_glue.c
View File

@ -24,8 +24,8 @@
#include "sha256_glue.h"
asmlinkage void sha256_block_data_order(u32 *digest, const void *data,
unsigned int num_blks);
asmlinkage void sha256_block_data_order(struct sha256_state *state,
const u8 *data, int num_blks);
int crypto_sha256_arm_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
@ -33,23 +33,20 @@ int crypto_sha256_arm_update(struct shash_desc *desc, const u8 *data,
/* make sure casting to sha256_block_fn() is safe */
BUILD_BUG_ON(offsetof(struct sha256_state, state) != 0);
return sha256_base_do_update(desc, data, len,
(sha256_block_fn *)sha256_block_data_order);
return sha256_base_do_update(desc, data, len, sha256_block_data_order);
}
EXPORT_SYMBOL(crypto_sha256_arm_update);
static int crypto_sha256_arm_final(struct shash_desc *desc, u8 *out)
{
sha256_base_do_finalize(desc,
(sha256_block_fn *)sha256_block_data_order);
sha256_base_do_finalize(desc, sha256_block_data_order);
return sha256_base_finish(desc, out);
}
int crypto_sha256_arm_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
sha256_base_do_update(desc, data, len,
(sha256_block_fn *)sha256_block_data_order);
sha256_base_do_update(desc, data, len, sha256_block_data_order);
return crypto_sha256_arm_final(desc, out);
}
EXPORT_SYMBOL(crypto_sha256_arm_finup);

12
arch/arm/crypto/sha512-glue.c
View File

@ -25,27 +25,25 @@ MODULE_ALIAS_CRYPTO("sha512");
MODULE_ALIAS_CRYPTO("sha384-arm");
MODULE_ALIAS_CRYPTO("sha512-arm");
asmlinkage void sha512_block_data_order(u64 *state, u8 const *src, int blocks);
asmlinkage void sha512_block_data_order(struct sha512_state *state,
u8 const *src, int blocks);
int sha512_arm_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
return sha512_base_do_update(desc, data, len,
(sha512_block_fn *)sha512_block_data_order);
return sha512_base_do_update(desc, data, len, sha512_block_data_order);
}
static int sha512_arm_final(struct shash_desc *desc, u8 *out)
{
sha512_base_do_finalize(desc,
(sha512_block_fn *)sha512_block_data_order);
sha512_base_do_finalize(desc, sha512_block_data_order);
return sha512_base_finish(desc, out);
}
int sha512_arm_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
sha512_base_do_update(desc, data, len,
(sha512_block_fn *)sha512_block_data_order);
sha512_base_do_update(desc, data, len, sha512_block_data_order);
return sha512_arm_final(desc, out);
}

1
arch/arm64/crypto/Kconfig
View File

@ -268,6 +268,7 @@ config CRYPTO_AES_ARM64_CE_CCM
depends on ARM64 && KERNEL_MODE_NEON
select CRYPTO_ALGAPI
select CRYPTO_AES_ARM64_CE
select CRYPTO_AES_ARM64_CE_BLK
select CRYPTO_AEAD
select CRYPTO_LIB_AES
help

265
arch/arm64/crypto/aes-ce-ccm-core.S
View File

@ -1,8 +1,11 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* aesce-ccm-core.S - AES-CCM transform for ARMv8 with Crypto Extensions
* aes-ce-ccm-core.S - AES-CCM transform for ARMv8 with Crypto Extensions
*
* Copyright (C) 2013 - 2017 Linaro Ltd <ard.biesheuvel@linaro.org>
* Copyright (C) 2013 - 2017 Linaro Ltd.
* Copyright (C) 2024 Google LLC
*
* Author: Ard Biesheuvel <ardb@kernel.org>
*/
#include <linux/linkage.h>
@ -11,211 +14,129 @@
.text
.arch armv8-a+crypto
/*
* u32 ce_aes_ccm_auth_data(u8 mac[], u8 const in[], u32 abytes,
* u32 macp, u8 const rk[], u32 rounds);
*/
SYM_FUNC_START(ce_aes_ccm_auth_data)
ld1 {v0.16b}, [x0] /* load mac */
cbz w3, 1f
sub w3, w3, #16
eor v1.16b, v1.16b, v1.16b
0: ldrb w7, [x1], #1 /* get 1 byte of input */
subs w2, w2, #1
add w3, w3, #1
ins v1.b[0], w7
ext v1.16b, v1.16b, v1.16b, #1 /* rotate in the input bytes */
beq 8f /* out of input? */
cbnz w3, 0b
eor v0.16b, v0.16b, v1.16b
1: ld1 {v3.4s}, [x4] /* load first round key */
prfm pldl1strm, [x1]
cmp w5, #12 /* which key size? */
add x6, x4, #16
sub w7, w5, #2 /* modified # of rounds */
bmi 2f
bne 5f
mov v5.16b, v3.16b
b 4f
2: mov v4.16b, v3.16b
ld1 {v5.4s}, [x6], #16 /* load 2nd round key */
3: aese v0.16b, v4.16b
aesmc v0.16b, v0.16b
4: ld1 {v3.4s}, [x6], #16 /* load next round key */
aese v0.16b, v5.16b
aesmc v0.16b, v0.16b
5: ld1 {v4.4s}, [x6], #16 /* load next round key */
subs w7, w7, #3
aese v0.16b, v3.16b
aesmc v0.16b, v0.16b
ld1 {v5.4s}, [x6], #16 /* load next round key */
bpl 3b
aese v0.16b, v4.16b
subs w2, w2, #16 /* last data? */
eor v0.16b, v0.16b, v5.16b /* final round */
bmi 6f
ld1 {v1.16b}, [x1], #16 /* load next input block */
eor v0.16b, v0.16b, v1.16b /* xor with mac */
bne 1b
6: st1 {v0.16b}, [x0] /* store mac */
beq 10f
adds w2, w2, #16
beq 10f
mov w3, w2
7: ldrb w7, [x1], #1
umov w6, v0.b[0]
eor w6, w6, w7
strb w6, [x0], #1
subs w2, w2, #1
beq 10f
ext v0.16b, v0.16b, v0.16b, #1 /* rotate out the mac bytes */
b 7b
8: cbz w3, 91f
mov w7, w3
add w3, w3, #16
9: ext v1.16b, v1.16b, v1.16b, #1
adds w7, w7, #1
bne 9b
91: eor v0.16b, v0.16b, v1.16b
st1 {v0.16b}, [x0]
10: mov w0, w3
ret
SYM_FUNC_END(ce_aes_ccm_auth_data)
.macro load_round_keys, rk, nr, tmp
sub w\tmp, \nr, #10
add \tmp, \rk, w\tmp, sxtw #4
ld1 {v10.4s-v13.4s}, [\rk]
ld1 {v14.4s-v17.4s}, [\tmp], #64
ld1 {v18.4s-v21.4s}, [\tmp], #64
ld1 {v3.4s-v5.4s}, [\tmp]
.endm
/*
* void ce_aes_ccm_final(u8 mac[], u8 const ctr[], u8 const rk[],
* u32 rounds);
*/
SYM_FUNC_START(ce_aes_ccm_final)
ld1 {v3.4s}, [x2], #16 /* load first round key */
ld1 {v0.16b}, [x0] /* load mac */
cmp w3, #12 /* which key size? */
sub w3, w3, #2 /* modified # of rounds */
ld1 {v1.16b}, [x1] /* load 1st ctriv */
bmi 0f
bne 3f
mov v5.16b, v3.16b
b 2f
0: mov v4.16b, v3.16b
1: ld1 {v5.4s}, [x2], #16 /* load next round key */
aese v0.16b, v4.16b
aesmc v0.16b, v0.16b
aese v1.16b, v4.16b
aesmc v1.16b, v1.16b
2: ld1 {v3.4s}, [x2], #16 /* load next round key */
aese v0.16b, v5.16b
aesmc v0.16b, v0.16b
aese v1.16b, v5.16b
aesmc v1.16b, v1.16b
3: ld1 {v4.4s}, [x2], #16 /* load next round key */
subs w3, w3, #3
aese v0.16b, v3.16b
aesmc v0.16b, v0.16b
aese v1.16b, v3.16b
aesmc v1.16b, v1.16b
bpl 1b
aese v0.16b, v4.16b
aese v1.16b, v4.16b
/* final round key cancels out */
eor v0.16b, v0.16b, v1.16b /* en-/decrypt the mac */
st1 {v0.16b}, [x0] /* store result */
ret
SYM_FUNC_END(ce_aes_ccm_final)
.macro dround, va, vb, vk
aese \va\().16b, \vk\().16b
aesmc \va\().16b, \va\().16b
aese \vb\().16b, \vk\().16b
aesmc \vb\().16b, \vb\().16b
.endm
.macro aes_encrypt, va, vb, nr
tbz \nr, #2, .L\@
dround \va, \vb, v10
dround \va, \vb, v11
tbz \nr, #1, .L\@
dround \va, \vb, v12
dround \va, \vb, v13
.L\@: .irp v, v14, v15, v16, v17, v18, v19, v20, v21, v3
dround \va, \vb, \v
.endr
aese \va\().16b, v4.16b
aese \vb\().16b, v4.16b
.endm
.macro aes_ccm_do_crypt,enc
cbz x2, 5f
ldr x8, [x6, #8] /* load lower ctr */
load_round_keys x3, w4, x10
ld1 {v0.16b}, [x5] /* load mac */
cbz x2, ce_aes_ccm_final
ldr x8, [x6, #8] /* load lower ctr */
CPU_LE( rev x8, x8 ) /* keep swabbed ctr in reg */
0: /* outer loop */
ld1 {v1.8b}, [x6] /* load upper ctr */
prfm pldl1strm, [x1]
add x8, x8, #1
rev x9, x8
cmp w4, #12 /* which key size? */
sub w7, w4, #2 /* get modified # of rounds */
ins v1.d[1], x9 /* no carry in lower ctr */
ld1 {v3.4s}, [x3] /* load first round key */
add x10, x3, #16
bmi 1f
bne 4f
mov v5.16b, v3.16b
b 3f
1: mov v4.16b, v3.16b
ld1 {v5.4s}, [x10], #16 /* load 2nd round key */
2: /* inner loop: 3 rounds, 2x interleaved */
aese v0.16b, v4.16b
aesmc v0.16b, v0.16b
aese v1.16b, v4.16b
aesmc v1.16b, v1.16b
3: ld1 {v3.4s}, [x10], #16 /* load next round key */
aese v0.16b, v5.16b
aesmc v0.16b, v0.16b
aese v1.16b, v5.16b
aesmc v1.16b, v1.16b
4: ld1 {v4.4s}, [x10], #16 /* load next round key */
subs w7, w7, #3
aese v0.16b, v3.16b
aesmc v0.16b, v0.16b
aese v1.16b, v3.16b
aesmc v1.16b, v1.16b
ld1 {v5.4s}, [x10], #16 /* load next round key */
bpl 2b
aese v0.16b, v4.16b
aese v1.16b, v4.16b
aes_encrypt v0, v1, w4
subs w2, w2, #16
bmi 6f /* partial block? */
bmi ce_aes_ccm_crypt_tail
ld1 {v2.16b}, [x1], #16 /* load next input block */
.if \enc == 1
eor v2.16b, v2.16b, v5.16b /* final round enc+mac */
eor v1.16b, v1.16b, v2.16b /* xor with crypted ctr */
eor v6.16b, v1.16b, v2.16b /* xor with crypted ctr */
.else
eor v2.16b, v2.16b, v1.16b /* xor with crypted ctr */
eor v1.16b, v2.16b, v5.16b /* final round enc */
eor v6.16b, v2.16b, v5.16b /* final round enc */
.endif
eor v0.16b, v0.16b, v2.16b /* xor mac with pt ^ rk[last] */
st1 {v1.16b}, [x0], #16 /* write output block */
st1 {v6.16b}, [x0], #16 /* write output block */
bne 0b
CPU_LE( rev x8, x8 )
st1 {v0.16b}, [x5] /* store mac */
str x8, [x6, #8] /* store lsb end of ctr (BE) */
5: ret
6: eor v0.16b, v0.16b, v5.16b /* final round mac */
eor v1.16b, v1.16b, v5.16b /* final round enc */
cbnz x7, ce_aes_ccm_final
st1 {v0.16b}, [x5] /* store mac */
add w2, w2, #16 /* process partial tail block */
7: ldrb w9, [x1], #1 /* get 1 byte of input */
umov w6, v1.b[0] /* get top crypted ctr byte */
umov w7, v0.b[0] /* get top mac byte */
.if \enc == 1
eor w7, w7, w9
eor w9, w9, w6
.else
eor w9, w9, w6
eor w7, w7, w9
.endif
strb w9, [x0], #1 /* store out byte */
strb w7, [x5], #1 /* store mac byte */
subs w2, w2, #1
beq 5b
ext v0.16b, v0.16b, v0.16b, #1 /* shift out mac byte */
ext v1.16b, v1.16b, v1.16b, #1 /* shift out ctr byte */
b 7b
ret
.endm
SYM_FUNC_START_LOCAL(ce_aes_ccm_crypt_tail)
eor v0.16b, v0.16b, v5.16b /* final round mac */
eor v1.16b, v1.16b, v5.16b /* final round enc */
add x1, x1, w2, sxtw /* rewind the input pointer (w2 < 0) */
add x0, x0, w2, sxtw /* rewind the output pointer */
adr_l x8, .Lpermute /* load permute vectors */
add x9, x8, w2, sxtw
sub x8, x8, w2, sxtw
ld1 {v7.16b-v8.16b}, [x9]
ld1 {v9.16b}, [x8]
ld1 {v2.16b}, [x1] /* load a full block of input */
tbl v1.16b, {v1.16b}, v7.16b /* move keystream to end of register */
eor v7.16b, v2.16b, v1.16b /* encrypt partial input block */
bif v2.16b, v7.16b, v22.16b /* select plaintext */
tbx v7.16b, {v6.16b}, v8.16b /* insert output from previous iteration */
tbl v2.16b, {v2.16b}, v9.16b /* copy plaintext to start of v2 */
eor v0.16b, v0.16b, v2.16b /* fold plaintext into mac */
st1 {v7.16b}, [x0] /* store output block */
cbz x7, 0f
SYM_INNER_LABEL(ce_aes_ccm_final, SYM_L_LOCAL)
ld1 {v1.16b}, [x7] /* load 1st ctriv */
aes_encrypt v0, v1, w4
/* final round key cancels out */
eor v0.16b, v0.16b, v1.16b /* en-/decrypt the mac */
0: st1 {v0.16b}, [x5] /* store result */
ret
SYM_FUNC_END(ce_aes_ccm_crypt_tail)
/*
* void ce_aes_ccm_encrypt(u8 out[], u8 const in[], u32 cbytes,
* u8 const rk[], u32 rounds, u8 mac[],
* u8 ctr[]);
* u8 ctr[], u8 const final_iv[]);
* void ce_aes_ccm_decrypt(u8 out[], u8 const in[], u32 cbytes,
* u8 const rk[], u32 rounds, u8 mac[],
* u8 ctr[]);
* u8 ctr[], u8 const final_iv[]);
*/
SYM_FUNC_START(ce_aes_ccm_encrypt)
movi v22.16b, #255
aes_ccm_do_crypt 1
SYM_FUNC_END(ce_aes_ccm_encrypt)
SYM_FUNC_START(ce_aes_ccm_decrypt)
movi v22.16b, #0
aes_ccm_do_crypt 0
SYM_FUNC_END(ce_aes_ccm_decrypt)
.section ".rodata", "a"
.align 6
.fill 15, 1, 0xff
.Lpermute:
.byte 0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7
.byte 0x8, 0x9, 0xa, 0xb, 0xc, 0xd, 0xe, 0xf
.fill 15, 1, 0xff

154
arch/arm64/crypto/aes-ce-ccm-glue.c
View File

@ -1,8 +1,11 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* aes-ccm-glue.c - AES-CCM transform for ARMv8 with Crypto Extensions
* aes-ce-ccm-glue.c - AES-CCM transform for ARMv8 with Crypto Extensions
*
* Copyright (C) 2013 - 2017 Linaro Ltd <ard.biesheuvel@linaro.org>
* Copyright (C) 2013 - 2017 Linaro Ltd.
* Copyright (C) 2024 Google LLC
*
* Author: Ard Biesheuvel <ardb@kernel.org>
*/
#include <asm/neon.h>
@ -15,6 +18,8 @@
#include "aes-ce-setkey.h"
MODULE_IMPORT_NS(CRYPTO_INTERNAL);
static int num_rounds(struct crypto_aes_ctx *ctx)
{
/*
@ -27,19 +32,17 @@ static int num_rounds(struct crypto_aes_ctx *ctx)
return 6 + ctx->key_length / 4;
}
asmlinkage u32 ce_aes_ccm_auth_data(u8 mac[], u8 const in[], u32 abytes,
u32 macp, u32 const rk[], u32 rounds);
asmlinkage u32 ce_aes_mac_update(u8 const in[], u32 const rk[], int rounds,
int blocks, u8 dg[], int enc_before,
int enc_after);
asmlinkage void ce_aes_ccm_encrypt(u8 out[], u8 const in[], u32 cbytes,
u32 const rk[], u32 rounds, u8 mac[],
u8 ctr[]);
u8 ctr[], u8 const final_iv[]);
asmlinkage void ce_aes_ccm_decrypt(u8 out[], u8 const in[], u32 cbytes,
u32 const rk[], u32 rounds, u8 mac[],
u8 ctr[]);
asmlinkage void ce_aes_ccm_final(u8 mac[], u8 const ctr[], u32 const rk[],
u32 rounds);
u8 ctr[], u8 const final_iv[]);
static int ccm_setkey(struct crypto_aead *tfm, const u8 *in_key,
unsigned int key_len)
@ -94,6 +97,41 @@ static int ccm_init_mac(struct aead_request *req, u8 maciv[], u32 msglen)
return 0;
}
static u32 ce_aes_ccm_auth_data(u8 mac[], u8 const in[], u32 abytes,
u32 macp, u32 const rk[], u32 rounds)
{
int enc_after = (macp + abytes) % AES_BLOCK_SIZE;
do {
u32 blocks = abytes / AES_BLOCK_SIZE;
if (macp == AES_BLOCK_SIZE || (!macp && blocks > 0)) {
u32 rem = ce_aes_mac_update(in, rk, rounds, blocks, mac,
macp, enc_after);
u32 adv = (blocks - rem) * AES_BLOCK_SIZE;
macp = enc_after ? 0 : AES_BLOCK_SIZE;
in += adv;
abytes -= adv;
if (unlikely(rem)) {
kernel_neon_end();
kernel_neon_begin();
macp = 0;
}
} else {
u32 l = min(AES_BLOCK_SIZE - macp, abytes);
crypto_xor(&mac[macp], in, l);
in += l;
macp += l;
abytes -= l;
}
} while (abytes > 0);
return macp;
}
static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[])
{
struct crypto_aead *aead = crypto_aead_reqtfm(req);
@ -101,7 +139,7 @@ static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[])
struct __packed { __be16 l; __be32 h; u16 len; } ltag;
struct scatter_walk walk;
u32 len = req->assoclen;
u32 macp = 0;
u32 macp = AES_BLOCK_SIZE;
/* prepend the AAD with a length tag */
if (len < 0xff00) {
@ -125,16 +163,11 @@ static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[])
scatterwalk_start(&walk, sg_next(walk.sg));
n = scatterwalk_clamp(&walk, len);
}
n = min_t(u32, n, SZ_4K); /* yield NEON at least every 4k */
p = scatterwalk_map(&walk);
macp = ce_aes_ccm_auth_data(mac, p, n, macp, ctx->key_enc,
num_rounds(ctx));
if (len / SZ_4K > (len - n) / SZ_4K) {
kernel_neon_end();
kernel_neon_begin();
}
len -= n;
scatterwalk_unmap(p);
@ -149,7 +182,7 @@ static int ccm_encrypt(struct aead_request *req)
struct crypto_aes_ctx *ctx = crypto_aead_ctx(aead);
struct skcipher_walk walk;
u8 __aligned(8) mac[AES_BLOCK_SIZE];
u8 buf[AES_BLOCK_SIZE];
u8 orig_iv[AES_BLOCK_SIZE];
u32 len = req->cryptlen;
int err;
@ -158,42 +191,55 @@ static int ccm_encrypt(struct aead_request *req)
return err;
/* preserve the original iv for the final round */
memcpy(buf, req->iv, AES_BLOCK_SIZE);
memcpy(orig_iv, req->iv, AES_BLOCK_SIZE);
err = skcipher_walk_aead_encrypt(&walk, req, false);
if (unlikely(err))
return err;
kernel_neon_begin();
if (req->assoclen)
ccm_calculate_auth_mac(req, mac);
while (walk.nbytes) {
do {
u32 tail = walk.nbytes % AES_BLOCK_SIZE;
bool final = walk.nbytes == walk.total;
const u8 *src = walk.src.virt.addr;
u8 *dst = walk.dst.virt.addr;
u8 buf[AES_BLOCK_SIZE];
u8 *final_iv = NULL;
if (final)
if (walk.nbytes == walk.total) {
tail = 0;
final_iv = orig_iv;
}
ce_aes_ccm_encrypt(walk.dst.virt.addr, walk.src.virt.addr,
walk.nbytes - tail, ctx->key_enc,
num_rounds(ctx), mac, walk.iv);
if (unlikely(walk.nbytes < AES_BLOCK_SIZE))
src = dst = memcpy(&buf[sizeof(buf) - walk.nbytes],
src, walk.nbytes);
if (!final)
kernel_neon_end();
err = skcipher_walk_done(&walk, tail);
if (!final)
kernel_neon_begin();
}
ce_aes_ccm_encrypt(dst, src, walk.nbytes - tail,
ctx->key_enc, num_rounds(ctx),
mac, walk.iv, final_iv);
ce_aes_ccm_final(mac, buf, ctx->key_enc, num_rounds(ctx));
if (unlikely(walk.nbytes < AES_BLOCK_SIZE))
memcpy(walk.dst.virt.addr, dst, walk.nbytes);
if (walk.nbytes) {
err = skcipher_walk_done(&walk, tail);
}
} while (walk.nbytes);
kernel_neon_end();
if (unlikely(err))
return err;
/* copy authtag to end of dst */
scatterwalk_map_and_copy(mac, req->dst, req->assoclen + req->cryptlen,
crypto_aead_authsize(aead), 1);
return err;
return 0;
}
static int ccm_decrypt(struct aead_request *req)
@ -203,7 +249,7 @@ static int ccm_decrypt(struct aead_request *req)
unsigned int authsize = crypto_aead_authsize(aead);
struct skcipher_walk walk;
u8 __aligned(8) mac[AES_BLOCK_SIZE];
u8 buf[AES_BLOCK_SIZE];
u8 orig_iv[AES_BLOCK_SIZE];
u32 len = req->cryptlen - authsize;
int err;
@ -212,34 +258,44 @@ static int ccm_decrypt(struct aead_request *req)
return err;
/* preserve the original iv for the final round */
memcpy(buf, req->iv, AES_BLOCK_SIZE);
memcpy(orig_iv, req->iv, AES_BLOCK_SIZE);
err = skcipher_walk_aead_decrypt(&walk, req, false);
if (unlikely(err))
return err;
kernel_neon_begin();
if (req->assoclen)
ccm_calculate_auth_mac(req, mac);
while (walk.nbytes) {
do {
u32 tail = walk.nbytes % AES_BLOCK_SIZE;
bool final = walk.nbytes == walk.total;
const u8 *src = walk.src.virt.addr;
u8 *dst = walk.dst.virt.addr;
u8 buf[AES_BLOCK_SIZE];
u8 *final_iv = NULL;
if (final)
if (walk.nbytes == walk.total) {
tail = 0;
final_iv = orig_iv;
}
ce_aes_ccm_decrypt(walk.dst.virt.addr, walk.src.virt.addr,
walk.nbytes - tail, ctx->key_enc,
num_rounds(ctx), mac, walk.iv);
if (unlikely(walk.nbytes < AES_BLOCK_SIZE))
src = dst = memcpy(&buf[sizeof(buf) - walk.nbytes],
src, walk.nbytes);
if (!final)
kernel_neon_end();
err = skcipher_walk_done(&walk, tail);
if (!final)
kernel_neon_begin();
}
ce_aes_ccm_decrypt(dst, src, walk.nbytes - tail,
ctx->key_enc, num_rounds(ctx),
mac, walk.iv, final_iv);
ce_aes_ccm_final(mac, buf, ctx->key_enc, num_rounds(ctx));
if (unlikely(walk.nbytes < AES_BLOCK_SIZE))
memcpy(walk.dst.virt.addr, dst, walk.nbytes);
if (walk.nbytes) {
err = skcipher_walk_done(&walk, tail);
}
} while (walk.nbytes);
kernel_neon_end();
@ -247,11 +303,11 @@ static int ccm_decrypt(struct aead_request *req)
return err;
/* compare calculated auth tag with the stored one */
scatterwalk_map_and_copy(buf, req->src,
scatterwalk_map_and_copy(orig_iv, req->src,
req->assoclen + req->cryptlen - authsize,
authsize, 0);
if (crypto_memneq(mac, buf, authsize))
if (crypto_memneq(mac, orig_iv, authsize))
return -EBADMSG;
return 0;
}
@ -290,6 +346,6 @@ module_init(aes_mod_init);
module_exit(aes_mod_exit);
MODULE_DESCRIPTION("Synchronous AES in CCM mode using ARMv8 Crypto Extensions");
MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>");
MODULE_AUTHOR("Ard Biesheuvel <ardb@kernel.org>");
MODULE_LICENSE("GPL v2");
MODULE_ALIAS_CRYPTO("ccm(aes)");

1
arch/arm64/crypto/aes-glue.c
View File

@ -1048,6 +1048,7 @@ unregister_ciphers:
#ifdef USE_V8_CRYPTO_EXTENSIONS
module_cpu_feature_match(AES, aes_init);
EXPORT_SYMBOL_NS(ce_aes_mac_update, CRYPTO_INTERNAL);
#else
module_init(aes_init);
EXPORT_SYMBOL(neon_aes_ecb_encrypt);

20
arch/powerpc/crypto/Kconfig
View File

@ -137,4 +137,24 @@ config CRYPTO_POLY1305_P10
- Power10 or later
- Little-endian
config CRYPTO_DEV_VMX
bool "Support for VMX cryptographic acceleration instructions"
depends on PPC64 && VSX
help
Support for VMX cryptographic acceleration instructions.
config CRYPTO_DEV_VMX_ENCRYPT
tristate "Encryption acceleration support on P8 CPU"
depends on CRYPTO_DEV_VMX
select CRYPTO_AES
select CRYPTO_CBC
select CRYPTO_CTR
select CRYPTO_GHASH
select CRYPTO_XTS
default m
help
Support for VMX cryptographic acceleration instructions on Power8 CPU.
This module supports acceleration for AES and GHASH in hardware. If you
choose 'M' here, this module will be called vmx-crypto.
endmenu

20
arch/powerpc/crypto/Makefile
View File

@ -16,6 +16,7 @@ obj-$(CONFIG_CRYPTO_VPMSUM_TESTER) += crc-vpmsum_test.o
obj-$(CONFIG_CRYPTO_AES_GCM_P10) += aes-gcm-p10-crypto.o
obj-$(CONFIG_CRYPTO_CHACHA20_P10) += chacha-p10-crypto.o
obj-$(CONFIG_CRYPTO_POLY1305_P10) += poly1305-p10-crypto.o
obj-$(CONFIG_CRYPTO_DEV_VMX_ENCRYPT) += vmx-crypto.o
aes-ppc-spe-y := aes-spe-core.o aes-spe-keys.o aes-tab-4k.o aes-spe-modes.o aes-spe-glue.o
md5-ppc-y := md5-asm.o md5-glue.o
@ -27,14 +28,29 @@ crct10dif-vpmsum-y := crct10dif-vpmsum_asm.o crct10dif-vpmsum_glue.o
aes-gcm-p10-crypto-y := aes-gcm-p10-glue.o aes-gcm-p10.o ghashp10-ppc.o aesp10-ppc.o
chacha-p10-crypto-y := chacha-p10-glue.o chacha-p10le-8x.o
poly1305-p10-crypto-y := poly1305-p10-glue.o poly1305-p10le_64.o
vmx-crypto-objs := vmx.o aesp8-ppc.o ghashp8-ppc.o aes.o aes_cbc.o aes_ctr.o aes_xts.o ghash.o
ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y)
override flavour := linux-ppc64le
else
ifdef CONFIG_PPC64_ELF_ABI_V2
override flavour := linux-ppc64-elfv2
else
override flavour := linux-ppc64
endif
endif
quiet_cmd_perl = PERL $@
cmd_perl = $(PERL) $< $(if $(CONFIG_CPU_LITTLE_ENDIAN), linux-ppc64le, linux-ppc64) > $@
cmd_perl = $(PERL) $< $(flavour) > $@
targets += aesp10-ppc.S ghashp10-ppc.S
targets += aesp10-ppc.S ghashp10-ppc.S aesp8-ppc.S ghashp8-ppc.S
$(obj)/aesp10-ppc.S $(obj)/ghashp10-ppc.S: $(obj)/%.S: $(src)/%.pl FORCE
$(call if_changed,perl)
$(obj)/aesp8-ppc.S $(obj)/ghashp8-ppc.S: $(obj)/%.S: $(src)/%.pl FORCE
$(call if_changed,perl)
OBJECT_FILES_NON_STANDARD_aesp10-ppc.o := y
OBJECT_FILES_NON_STANDARD_ghashp10-ppc.o := y
OBJECT_FILES_NON_STANDARD_aesp8-ppc.o := y

0
drivers/crypto/vmx/aes.c → arch/powerpc/crypto/aes.c
View File

0
drivers/crypto/vmx/aes_cbc.c → arch/powerpc/crypto/aes_cbc.c
View File

0
drivers/crypto/vmx/aes_ctr.c → arch/powerpc/crypto/aes_ctr.c
View File

0
drivers/crypto/vmx/aes_xts.c → arch/powerpc/crypto/aes_xts.c
View File

0
drivers/crypto/vmx/aesp8-ppc.h → arch/powerpc/crypto/aesp8-ppc.h
View File

0
drivers/crypto/vmx/aesp8-ppc.pl → arch/powerpc/crypto/aesp8-ppc.pl
View File

0
drivers/crypto/vmx/ghash.c → arch/powerpc/crypto/ghash.c
View File

0
drivers/crypto/vmx/ghashp8-ppc.pl → arch/powerpc/crypto/ghashp8-ppc.pl
View File

0
drivers/crypto/vmx/vmx.c → arch/powerpc/crypto/vmx.c
View File

5
crypto/Kconfig
View File

@ -1269,10 +1269,11 @@ config CRYPTO_JITTERENTROPY
A non-physical non-deterministic ("true") RNG (e.g., an entropy source
compliant with NIST SP800-90B) intended to provide a seed to a
deterministic RNG (e.g. per NIST SP800-90C).
deterministic RNG (e.g., per NIST SP800-90C).
This RNG does not perform any cryptographic whitening of the generated
random numbers.
See https://www.chronox.de/jent.html
See https://www.chronox.de/jent/
if CRYPTO_JITTERENTROPY
if CRYPTO_FIPS && EXPERT

21
crypto/ahash.c
View File

@ -618,6 +618,16 @@ int crypto_has_ahash(const char *alg_name, u32 type, u32 mask)
}
EXPORT_SYMBOL_GPL(crypto_has_ahash);
static bool crypto_hash_alg_has_setkey(struct hash_alg_common *halg)
{
struct crypto_alg *alg = &halg->base;
if (alg->cra_type == &crypto_shash_type)
return crypto_shash_alg_has_setkey(__crypto_shash_alg(alg));
return __crypto_ahash_alg(alg)->setkey != ahash_nosetkey;
}
struct crypto_ahash *crypto_clone_ahash(struct crypto_ahash *hash)
{
struct hash_alg_common *halg = crypto_hash_alg_common(hash);
@ -760,16 +770,5 @@ int ahash_register_instance(struct crypto_template *tmpl,
}
EXPORT_SYMBOL_GPL(ahash_register_instance);
bool crypto_hash_alg_has_setkey(struct hash_alg_common *halg)
{
struct crypto_alg *alg = &halg->base;
if (alg->cra_type == &crypto_shash_type)
return crypto_shash_alg_has_setkey(__crypto_shash_alg(alg));
return __crypto_ahash_alg(alg)->setkey != ahash_nosetkey;
}
EXPORT_SYMBOL_GPL(crypto_hash_alg_has_setkey);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Asynchronous cryptographic hash type");

4
crypto/asymmetric_keys/verify_pefile.c
View File

@ -28,7 +28,7 @@ static int pefile_parse_binary(const void *pebuf, unsigned int pelen,
const struct pe32plus_opt_hdr *pe64;
const struct data_directory *ddir;
const struct data_dirent *dde;
const struct section_header *secs, *sec;
const struct section_header *sec;
size_t cursor, datalen = pelen;
kenter("");
@ -110,7 +110,7 @@ static int pefile_parse_binary(const void *pebuf, unsigned int pelen,
ctx->n_sections = pe->sections;
if (ctx->n_sections > (ctx->header_size - cursor) / sizeof(*sec))
return -ELIBBAD;
ctx->secs = secs = pebuf + cursor;
ctx->secs = pebuf + cursor;
return 0;
}

63
crypto/dh.c
View File

@ -106,6 +106,12 @@ err_clear_ctx:
*/
static int dh_is_pubkey_valid(struct dh_ctx *ctx, MPI y)
{
MPI val, q;
int ret;
if (!fips_enabled)
return 0;
if (unlikely(!ctx->p))
return -EINVAL;
@ -125,41 +131,36 @@ static int dh_is_pubkey_valid(struct dh_ctx *ctx, MPI y)
*
* For the safe-prime groups q = (p - 1)/2.
*/
if (fips_enabled) {
MPI val, q;
int ret;
val = mpi_alloc(0);
if (!val)
return -ENOMEM;
q = mpi_alloc(mpi_get_nlimbs(ctx->p));
if (!q) {
mpi_free(val);
return -ENOMEM;
}
/*
* ->p is odd, so no need to explicitly subtract one
* from it before shifting to the right.
*/
mpi_rshift(q, ctx->p, 1);
ret = mpi_powm(val, y, q, ctx->p);
mpi_free(q);
if (ret) {
mpi_free(val);
return ret;
}
ret = mpi_cmp_ui(val, 1);
val = mpi_alloc(0);
if (!val)
return -ENOMEM;
q = mpi_alloc(mpi_get_nlimbs(ctx->p));
if (!q) {
mpi_free(val);
if (ret != 0)
return -EINVAL;
return -ENOMEM;
}
/*
* ->p is odd, so no need to explicitly subtract one
* from it before shifting to the right.
*/
mpi_rshift(q, ctx->p, 1);
ret = mpi_powm(val, y, q, ctx->p);
mpi_free(q);
if (ret) {
mpi_free(val);
return ret;
}
ret = mpi_cmp_ui(val, 1);
mpi_free(val);
if (ret != 0)
return -EINVAL;
return 0;
}

4
crypto/pcbc.c
View File

@ -71,7 +71,7 @@ static int crypto_pcbc_encrypt(struct skcipher_request *req)
err = skcipher_walk_virt(&walk, req, false);
while ((nbytes = walk.nbytes)) {
while (walk.nbytes) {
if (walk.src.virt.addr == walk.dst.virt.addr)
nbytes = crypto_pcbc_encrypt_inplace(req, &walk,
cipher);
@ -138,7 +138,7 @@ static int crypto_pcbc_decrypt(struct skcipher_request *req)
err = skcipher_walk_virt(&walk, req, false);
while ((nbytes = walk.nbytes)) {
while (walk.nbytes) {
if (walk.src.virt.addr == walk.dst.virt.addr)
nbytes = crypto_pcbc_decrypt_inplace(req, &walk,
cipher);

36
crypto/rsa.c
View File

@ -24,14 +24,38 @@ struct rsa_mpi_key {
MPI qinv;
};
static int rsa_check_payload(MPI x, MPI n)
{
MPI n1;
if (mpi_cmp_ui(x, 1) <= 0)
return -EINVAL;
n1 = mpi_alloc(0);
if (!n1)
return -ENOMEM;
if (mpi_sub_ui(n1, n, 1) || mpi_cmp(x, n1) >= 0) {
mpi_free(n1);
return -EINVAL;
}
mpi_free(n1);
return 0;
}
/*
* RSAEP function [RFC3447 sec 5.1.1]
* c = m^e mod n;
*/
static int _rsa_enc(const struct rsa_mpi_key *key, MPI c, MPI m)
{
/* (1) Validate 0 <= m < n */
if (mpi_cmp_ui(m, 0) < 0 || mpi_cmp(m, key->n) >= 0)
/*
* Even though (1) in RFC3447 only requires 0 <= m <= n - 1, we are
* slightly more conservative and require 1 < m < n - 1. This is in line
* with SP 800-56Br2, Section 7.1.1.
*/
if (rsa_check_payload(m, key->n))
return -EINVAL;
/* (2) c = m^e mod n */
@ -50,8 +74,12 @@ static int _rsa_dec_crt(const struct rsa_mpi_key *key, MPI m_or_m1_or_h, MPI c)
MPI m2, m12_or_qh;
int ret = -ENOMEM;
/* (1) Validate 0 <= c < n */
if (mpi_cmp_ui(c, 0) < 0 || mpi_cmp(c, key->n) >= 0)
/*
* Even though (1) in RFC3447 only requires 0 <= c <= n - 1, we are
* slightly more conservative and require 1 < c < n - 1. This is in line
* with SP 800-56Br2, Section 7.1.2.
*/
if (rsa_check_payload(c, key->n))
return -EINVAL;
m2 = mpi_alloc(0);

38
crypto/scompress.c
View File

@ -117,6 +117,7 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir)
struct crypto_scomp *scomp = *tfm_ctx;
void **ctx = acomp_request_ctx(req);
struct scomp_scratch *scratch;
void *src, *dst;
unsigned int dlen;
int ret;
@ -134,13 +135,25 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir)
scratch = raw_cpu_ptr(&scomp_scratch);
spin_lock(&scratch->lock);
scatterwalk_map_and_copy(scratch->src, req->src, 0, req->slen, 0);
if (dir)
ret = crypto_scomp_compress(scomp, scratch->src, req->slen,
scratch->dst, &req->dlen, *ctx);
if (sg_nents(req->src) == 1 && !PageHighMem(sg_page(req->src))) {
src = page_to_virt(sg_page(req->src)) + req->src->offset;
} else {
scatterwalk_map_and_copy(scratch->src, req->src, 0,
req->slen, 0);
src = scratch->src;
}
if (req->dst && sg_nents(req->dst) == 1 && !PageHighMem(sg_page(req->dst)))
dst = page_to_virt(sg_page(req->dst)) + req->dst->offset;
else
ret = crypto_scomp_decompress(scomp, scratch->src, req->slen,
scratch->dst, &req->dlen, *ctx);
dst = scratch->dst;
if (dir)
ret = crypto_scomp_compress(scomp, src, req->slen,
dst, &req->dlen, *ctx);
else
ret = crypto_scomp_decompress(scomp, src, req->slen,
dst, &req->dlen, *ctx);
if (!ret) {
if (!req->dst) {
req->dst = sgl_alloc(req->dlen, GFP_ATOMIC, NULL);
@ -152,8 +165,17 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir)
ret = -ENOSPC;
goto out;
}
scatterwalk_map_and_copy(scratch->dst, req->dst, 0, req->dlen,
1);
if (dst == scratch->dst) {
scatterwalk_map_and_copy(scratch->dst, req->dst, 0,
req->dlen, 1);
} else {
int nr_pages = DIV_ROUND_UP(req->dst->offset + req->dlen, PAGE_SIZE);
int i;
struct page *dst_page = sg_page(req->dst);
for (i = 0; i < nr_pages; i++)
flush_dcache_page(dst_page + i);
}
}
out:
spin_unlock(&scratch->lock);

3
crypto/tcrypt.c
View File

@ -1851,6 +1851,9 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb)
ret = min(ret, tcrypt_test("cbc(aria)"));
ret = min(ret, tcrypt_test("ctr(aria)"));
break;
case 193:
ret = min(ret, tcrypt_test("ffdhe2048(dh)"));
break;
case 200:
test_cipher_speed("ecb(aes)", ENCRYPT, sec, NULL, 0,
speed_template_16_24_32);

8
crypto/testmgr.c
View File

@ -5720,14 +5720,6 @@ static const struct alg_test_desc alg_test_descs[] = {
}
}, {
#endif
.alg = "xts4096(paes)",
.test = alg_test_null,
.fips_allowed = 1,
}, {
.alg = "xts512(paes)",
.test = alg_test_null,
.fips_allowed = 1,
}, {
.alg = "xxhash64",
.test = alg_test_hash,
.fips_allowed = 1,

6
drivers/char/hw_random/hisi-rng.c
View File

@ -89,10 +89,8 @@ static int hisi_rng_probe(struct platform_device *pdev)
rng->rng.read = hisi_rng_read;
ret = devm_hwrng_register(&pdev->dev, &rng->rng);
if (ret) {
dev_err(&pdev->dev, "failed to register hwrng\n");
return ret;
}
if (ret)
return dev_err_probe(&pdev->dev, ret, "failed to register hwrng\n");
return 0;
}

14
drivers/crypto/Kconfig
View File

@ -611,13 +611,13 @@ config CRYPTO_DEV_QCOM_RNG
To compile this driver as a module, choose M here. The
module will be called qcom-rng. If unsure, say N.
config CRYPTO_DEV_VMX
bool "Support for VMX cryptographic acceleration instructions"
depends on PPC64 && VSX
help
Support for VMX cryptographic acceleration instructions.
source "drivers/crypto/vmx/Kconfig"
#config CRYPTO_DEV_VMX
# bool "Support for VMX cryptographic acceleration instructions"
# depends on PPC64 && VSX
# help
# Support for VMX cryptographic acceleration instructions.
#
#source "drivers/crypto/vmx/Kconfig"
config CRYPTO_DEV_IMGTEC_HASH
tristate "Imagination Technologies hardware hash accelerator"

2
drivers/crypto/Makefile
View File

@ -42,7 +42,7 @@ obj-$(CONFIG_CRYPTO_DEV_SL3516) += gemini/
obj-y += stm32/
obj-$(CONFIG_CRYPTO_DEV_TALITOS) += talitos.o
obj-$(CONFIG_CRYPTO_DEV_VIRTIO) += virtio/
obj-$(CONFIG_CRYPTO_DEV_VMX) += vmx/
#obj-$(CONFIG_CRYPTO_DEV_VMX) += vmx/
obj-$(CONFIG_CRYPTO_DEV_BCM_SPU) += bcm/
obj-$(CONFIG_CRYPTO_DEV_SAFEXCEL) += inside-secure/
obj-$(CONFIG_CRYPTO_DEV_ARTPEC6) += axis/

2
drivers/crypto/allwinner/sun8i-ce/sun8i-ce-hash.c
View File

@ -362,7 +362,7 @@ int sun8i_ce_hash_run(struct crypto_engine *engine, void *breq)
digestsize = SHA512_DIGEST_SIZE;
/* the padding could be up to two block. */
buf = kzalloc(bs * 2, GFP_KERNEL | GFP_DMA);
buf = kcalloc(2, bs, GFP_KERNEL | GFP_DMA);
if (!buf) {
err = -ENOMEM;
goto theend;

11
drivers/crypto/ccp/platform-access.c
View File

@ -118,9 +118,16 @@ int psp_send_platform_access_msg(enum psp_platform_access_msg msg,
goto unlock;
}
/* Store the status in request header for caller to investigate */
/*
* Read status from PSP. If status is non-zero, it indicates an error
* occurred during "processing" of the command.
* If status is zero, it indicates the command was "processed"
* successfully, but the result of the command is in the payload.
* Return both cases to the caller as -EIO to investigate.
*/
cmd_reg = ioread32(cmd);
req->header.status = FIELD_GET(PSP_CMDRESP_STS, cmd_reg);
if (FIELD_GET(PSP_CMDRESP_STS, cmd_reg))
req->header.status = FIELD_GET(PSP_CMDRESP_STS, cmd_reg);
if (req->header.status) {
ret = -EIO;
goto unlock;

11
drivers/crypto/ccp/psp-dev.c
View File

@ -156,11 +156,14 @@ static unsigned int psp_get_capability(struct psp_device *psp)
}
psp->capability = val;
/* Detect if TSME and SME are both enabled */
/* Detect TSME and/or SME status */
if (PSP_CAPABILITY(psp, PSP_SECURITY_REPORTING) &&
psp->capability & (PSP_SECURITY_TSME_STATUS << PSP_CAPABILITY_PSP_SECURITY_OFFSET) &&
cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT))
dev_notice(psp->dev, "psp: Both TSME and SME are active, SME is unnecessary when TSME is active.\n");
psp->capability & (PSP_SECURITY_TSME_STATUS << PSP_CAPABILITY_PSP_SECURITY_OFFSET)) {
if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT))
dev_notice(psp->dev, "psp: Both TSME and SME are active, SME is unnecessary when TSME is active.\n");
else
dev_notice(psp->dev, "psp: TSME enabled\n");
}
return 0;
}

58
drivers/crypto/hisilicon/debugfs.c
View File

@ -24,6 +24,8 @@
#define QM_DFX_QN_SHIFT 16
#define QM_DFX_CNT_CLR_CE 0x100118
#define QM_DBG_WRITE_LEN 1024
#define QM_IN_IDLE_ST_REG 0x1040e4
#define QM_IN_IDLE_STATE 0x1
static const char * const qm_debug_file_name[] = {
[CURRENT_QM] = "current_qm",
@ -81,6 +83,30 @@ static const struct debugfs_reg32 qm_dfx_regs[] = {
{"QM_DFX_FF_ST5 ", 0x1040dc},
{"QM_DFX_FF_ST6 ", 0x1040e0},
{"QM_IN_IDLE_ST ", 0x1040e4},
{"QM_CACHE_CTL ", 0x100050},
{"QM_TIMEOUT_CFG ", 0x100070},
{"QM_DB_TIMEOUT_CFG ", 0x100074},
{"QM_FLR_PENDING_TIME_CFG ", 0x100078},
{"QM_ARUSR_MCFG1 ", 0x100088},
{"QM_AWUSR_MCFG1 ", 0x100098},
{"QM_AXI_M_CFG_ENABLE ", 0x1000B0},
{"QM_RAS_CE_THRESHOLD ", 0x1000F8},
{"QM_AXI_TIMEOUT_CTRL ", 0x100120},
{"QM_AXI_TIMEOUT_STATUS ", 0x100124},
{"QM_CQE_AGGR_TIMEOUT_CTRL ", 0x100144},
{"ACC_RAS_MSI_INT_SEL ", 0x1040fc},
{"QM_CQE_OUT ", 0x104100},
{"QM_EQE_OUT ", 0x104104},
{"QM_AEQE_OUT ", 0x104108},
{"QM_DB_INFO0 ", 0x104180},
{"QM_DB_INFO1 ", 0x104184},
{"QM_AM_CTRL_GLOBAL ", 0x300000},
{"QM_AM_CURR_PORT_STS ", 0x300100},
{"QM_AM_CURR_TRANS_RETURN ", 0x300150},
{"QM_AM_CURR_RD_MAX_TXID ", 0x300154},
{"QM_AM_CURR_WR_MAX_TXID ", 0x300158},
{"QM_AM_ALARM_RRESP ", 0x300180},
{"QM_AM_ALARM_BRESP ", 0x300184},
};
static const struct debugfs_reg32 qm_vf_dfx_regs[] = {
@ -1001,6 +1027,30 @@ static int qm_diff_regs_show(struct seq_file *s, void *unused)
}
DEFINE_SHOW_ATTRIBUTE(qm_diff_regs);
static int qm_state_show(struct seq_file *s, void *unused)
{
struct hisi_qm *qm = s->private;
u32 val;
int ret;
/* If device is in suspended, directly return the idle state. */
ret = hisi_qm_get_dfx_access(qm);
if (!ret) {
val = readl(qm->io_base + QM_IN_IDLE_ST_REG);
hisi_qm_put_dfx_access(qm);
} else if (ret == -EAGAIN) {
val = QM_IN_IDLE_STATE;
} else {
return ret;
}
seq_printf(s, "%u\n", val);
return 0;
}
DEFINE_SHOW_ATTRIBUTE(qm_state);
static ssize_t qm_status_read(struct file *filp, char __user *buffer,
size_t count, loff_t *pos)
{
@ -1062,6 +1112,7 @@ DEFINE_DEBUGFS_ATTRIBUTE(qm_atomic64_ops, qm_debugfs_atomic64_get,
void hisi_qm_debug_init(struct hisi_qm *qm)
{
struct dfx_diff_registers *qm_regs = qm->debug.qm_diff_regs;
struct qm_dev_dfx *dev_dfx = &qm->debug.dev_dfx;
struct qm_dfx *dfx = &qm->debug.dfx;
struct dentry *qm_d;
void *data;
@ -1072,6 +1123,9 @@ void hisi_qm_debug_init(struct hisi_qm *qm)
/* only show this in PF */
if (qm->fun_type == QM_HW_PF) {
debugfs_create_file("qm_state", 0444, qm->debug.qm_d,
qm, &qm_state_fops);
qm_create_debugfs_file(qm, qm->debug.debug_root, CURRENT_QM);
for (i = CURRENT_Q; i < DEBUG_FILE_NUM; i++)
qm_create_debugfs_file(qm, qm->debug.qm_d, i);
@ -1087,6 +1141,10 @@ void hisi_qm_debug_init(struct hisi_qm *qm)
debugfs_create_file("status", 0444, qm->debug.qm_d, qm,
&qm_status_fops);
debugfs_create_u32("dev_state", 0444, qm->debug.qm_d, &dev_dfx->dev_state);
debugfs_create_u32("dev_timeout", 0644, qm->debug.qm_d, &dev_dfx->dev_timeout);
for (i = 0; i < ARRAY_SIZE(qm_dfx_files); i++) {
data = (atomic64_t *)((uintptr_t)dfx + qm_dfx_files[i].offset);
debugfs_create_file(qm_dfx_files[i].name,

2
drivers/crypto/hisilicon/hpre/hpre_main.c
View File

@ -440,7 +440,7 @@ MODULE_PARM_DESC(vfs_num, "Number of VFs to enable(1-63), 0(default)");
struct hisi_qp *hpre_create_qp(u8 type)
{
int node = cpu_to_node(smp_processor_id());
int node = cpu_to_node(raw_smp_processor_id());
struct hisi_qp *qp = NULL;
int ret;

184
drivers/crypto/hisilicon/qm.c
View File

@ -236,6 +236,12 @@
#define QM_DEV_ALG_MAX_LEN 256
/* abnormal status value for stopping queue */
#define QM_STOP_QUEUE_FAIL 1
#define QM_DUMP_SQC_FAIL 3
#define QM_DUMP_CQC_FAIL 4
#define QM_FINISH_WAIT 5
#define QM_MK_CQC_DW3_V1(hop_num, pg_sz, buf_sz, cqe_sz) \
(((hop_num) << QM_CQ_HOP_NUM_SHIFT) | \
((pg_sz) << QM_CQ_PAGE_SIZE_SHIFT) | \
@ -312,6 +318,7 @@ static const struct hisi_qm_cap_info qm_cap_info_comm[] = {
{QM_SUPPORT_DB_ISOLATION, 0x30, 0, BIT(0), 0x0, 0x0, 0x0},
{QM_SUPPORT_FUNC_QOS, 0x3100, 0, BIT(8), 0x0, 0x0, 0x1},
{QM_SUPPORT_STOP_QP, 0x3100, 0, BIT(9), 0x0, 0x0, 0x1},
{QM_SUPPORT_STOP_FUNC, 0x3100, 0, BIT(10), 0x0, 0x0, 0x1},
{QM_SUPPORT_MB_COMMAND, 0x3100, 0, BIT(11), 0x0, 0x0, 0x1},
{QM_SUPPORT_SVA_PREFETCH, 0x3100, 0, BIT(14), 0x0, 0x0, 0x1},
};
@ -1674,6 +1681,11 @@ unlock:
return ret;
}
static int qm_drain_qm(struct hisi_qm *qm)
{
return hisi_qm_mb(qm, QM_MB_CMD_FLUSH_QM, 0, 0, 0);
}
static int qm_stop_qp(struct hisi_qp *qp)
{
return hisi_qm_mb(qp->qm, QM_MB_CMD_STOP_QP, 0, qp->qp_id, 0);
@ -2031,43 +2043,25 @@ static void qp_stop_fail_cb(struct hisi_qp *qp)
}
}
/**
* qm_drain_qp() - Drain a qp.
* @qp: The qp we want to drain.
*
* Determine whether the queue is cleared by judging the tail pointers of
* sq and cq.
*/
static int qm_drain_qp(struct hisi_qp *qp)
static int qm_wait_qp_empty(struct hisi_qm *qm, u32 *state, u32 qp_id)
{
struct hisi_qm *qm = qp->qm;
struct device *dev = &qm->pdev->dev;
struct qm_sqc sqc;
struct qm_cqc cqc;
int ret, i = 0;
/* No need to judge if master OOO is blocked. */
if (qm_check_dev_error(qm))
return 0;
/* Kunpeng930 supports drain qp by device */
if (test_bit(QM_SUPPORT_STOP_QP, &qm->caps)) {
ret = qm_stop_qp(qp);
if (ret)
dev_err(dev, "Failed to stop qp(%u)!\n", qp->qp_id);
return ret;
}
while (++i) {
ret = qm_set_and_get_xqc(qm, QM_MB_CMD_SQC, &sqc, qp->qp_id, 1);
ret = qm_set_and_get_xqc(qm, QM_MB_CMD_SQC, &sqc, qp_id, 1);
if (ret) {
dev_err_ratelimited(dev, "Failed to dump sqc!\n");
*state = QM_DUMP_SQC_FAIL;
return ret;
}
ret = qm_set_and_get_xqc(qm, QM_MB_CMD_CQC, &cqc, qp->qp_id, 1);
ret = qm_set_and_get_xqc(qm, QM_MB_CMD_CQC, &cqc, qp_id, 1);
if (ret) {
dev_err_ratelimited(dev, "Failed to dump cqc!\n");
*state = QM_DUMP_CQC_FAIL;
return ret;
}
@ -2076,8 +2070,9 @@ static int qm_drain_qp(struct hisi_qp *qp)
break;
if (i == MAX_WAIT_COUNTS) {
dev_err(dev, "Fail to empty queue %u!\n", qp->qp_id);
return -EBUSY;
dev_err(dev, "Fail to empty queue %u!\n", qp_id);
*state = QM_STOP_QUEUE_FAIL;
return -ETIMEDOUT;
}
usleep_range(WAIT_PERIOD_US_MIN, WAIT_PERIOD_US_MAX);
@ -2086,9 +2081,53 @@ static int qm_drain_qp(struct hisi_qp *qp)
return 0;
}
static int qm_stop_qp_nolock(struct hisi_qp *qp)
/**
* qm_drain_qp() - Drain a qp.
* @qp: The qp we want to drain.
*
* If the device does not support stopping queue by sending mailbox,
* determine whether the queue is cleared by judging the tail pointers of
* sq and cq.
*/
static int qm_drain_qp(struct hisi_qp *qp)
{
struct device *dev = &qp->qm->pdev->dev;
struct hisi_qm *qm = qp->qm;
struct hisi_qm *pf_qm = pci_get_drvdata(pci_physfn(qm->pdev));
u32 state = 0;
int ret;
/* No need to judge if master OOO is blocked. */
if (qm_check_dev_error(pf_qm))
return 0;
/* HW V3 supports drain qp by device */
if (test_bit(QM_SUPPORT_STOP_QP, &qm->caps)) {
ret = qm_stop_qp(qp);
if (ret) {
dev_err(&qm->pdev->dev, "Failed to stop qp!\n");
state = QM_STOP_QUEUE_FAIL;
goto set_dev_state;
}
return ret;
}
ret = qm_wait_qp_empty(qm, &state, qp->qp_id);
if (ret)
goto set_dev_state;
return 0;
set_dev_state:
if (qm->debug.dev_dfx.dev_timeout)
qm->debug.dev_dfx.dev_state = state;
return ret;
}
static void qm_stop_qp_nolock(struct hisi_qp *qp)
{
struct hisi_qm *qm = qp->qm;
struct device *dev = &qm->pdev->dev;
int ret;
/*
@ -2099,39 +2138,36 @@ static int qm_stop_qp_nolock(struct hisi_qp *qp)
*/
if (atomic_read(&qp->qp_status.flags) != QP_START) {
qp->is_resetting = false;
return 0;
return;
}
atomic_set(&qp->qp_status.flags, QP_STOP);
ret = qm_drain_qp(qp);
if (ret)
dev_err(dev, "Failed to drain out data for stopping!\n");
/* V3 supports direct stop function when FLR prepare */
if (qm->ver < QM_HW_V3 || qm->status.stop_reason == QM_NORMAL) {
ret = qm_drain_qp(qp);
if (ret)
dev_err(dev, "Failed to drain out data for stopping qp(%u)!\n", qp->qp_id);
}
flush_workqueue(qp->qm->wq);
flush_workqueue(qm->wq);
if (unlikely(qp->is_resetting && atomic_read(&qp->qp_status.used)))
qp_stop_fail_cb(qp);
dev_dbg(dev, "stop queue %u!", qp->qp_id);
return 0;
}
/**
* hisi_qm_stop_qp() - Stop a qp in qm.
* @qp: The qp we want to stop.
*
* This function is reverse of hisi_qm_start_qp. Return 0 if successful.
* This function is reverse of hisi_qm_start_qp.
*/
int hisi_qm_stop_qp(struct hisi_qp *qp)
void hisi_qm_stop_qp(struct hisi_qp *qp)
{
int ret;
down_write(&qp->qm->qps_lock);
ret = qm_stop_qp_nolock(qp);
qm_stop_qp_nolock(qp);
up_write(&qp->qm->qps_lock);
return ret;
}
EXPORT_SYMBOL_GPL(hisi_qm_stop_qp);
@ -2309,7 +2345,31 @@ static int hisi_qm_uacce_start_queue(struct uacce_queue *q)
static void hisi_qm_uacce_stop_queue(struct uacce_queue *q)
{
hisi_qm_stop_qp(q->priv);
struct hisi_qp *qp = q->priv;
struct hisi_qm *qm = qp->qm;
struct qm_dev_dfx *dev_dfx = &qm->debug.dev_dfx;
u32 i = 0;
hisi_qm_stop_qp(qp);
if (!dev_dfx->dev_timeout || !dev_dfx->dev_state)
return;
/*
* After the queue fails to be stopped,
* wait for a period of time before releasing the queue.
*/
while (++i) {
msleep(WAIT_PERIOD);
/* Since dev_timeout maybe modified, check i >= dev_timeout */
if (i >= dev_dfx->dev_timeout) {
dev_err(&qm->pdev->dev, "Stop q %u timeout, state %u\n",
qp->qp_id, dev_dfx->dev_state);
dev_dfx->dev_state = QM_FINISH_WAIT;
break;
}
}
}
static int hisi_qm_is_q_updated(struct uacce_queue *q)
@ -3054,25 +3114,18 @@ static int qm_restart(struct hisi_qm *qm)
}
/* Stop started qps in reset flow */
static int qm_stop_started_qp(struct hisi_qm *qm)
static void qm_stop_started_qp(struct hisi_qm *qm)
{
struct device *dev = &qm->pdev->dev;
struct hisi_qp *qp;
int i, ret;
int i;
for (i = 0; i < qm->qp_num; i++) {
qp = &qm->qp_array[i];
if (qp && atomic_read(&qp->qp_status.flags) == QP_START) {
if (atomic_read(&qp->qp_status.flags) == QP_START) {
qp->is_resetting = true;
ret = qm_stop_qp_nolock(qp);
if (ret < 0) {
dev_err(dev, "Failed to stop qp%d!\n", i);
return ret;
}
qm_stop_qp_nolock(qp);
}
}
return 0;
}
/**
@ -3112,21 +3165,31 @@ int hisi_qm_stop(struct hisi_qm *qm, enum qm_stop_reason r)
down_write(&qm->qps_lock);
qm->status.stop_reason = r;
if (atomic_read(&qm->status.flags) == QM_STOP)
goto err_unlock;
/* Stop all the request sending at first. */
atomic_set(&qm->status.flags, QM_STOP);
qm->status.stop_reason = r;
if (qm->status.stop_reason == QM_SOFT_RESET ||
qm->status.stop_reason == QM_DOWN) {
if (qm->status.stop_reason != QM_NORMAL) {
hisi_qm_set_hw_reset(qm, QM_RESET_STOP_TX_OFFSET);
ret = qm_stop_started_qp(qm);
if (ret < 0) {
dev_err(dev, "Failed to stop started qp!\n");
goto err_unlock;
/*
* When performing soft reset, the hardware will no longer
* do tasks, and the tasks in the device will be flushed
* out directly since the master ooo is closed.
*/
if (test_bit(QM_SUPPORT_STOP_FUNC, &qm->caps) &&
r != QM_SOFT_RESET) {
ret = qm_drain_qm(qm);
if (ret) {
dev_err(dev, "failed to drain qm!\n");
goto err_unlock;
}
}
qm_stop_started_qp(qm);
hisi_qm_set_hw_reset(qm, QM_RESET_STOP_RX_OFFSET);
}
@ -3141,6 +3204,7 @@ int hisi_qm_stop(struct hisi_qm *qm, enum qm_stop_reason r)
}
qm_clear_queues(qm);
qm->status.stop_reason = QM_NORMAL;
err_unlock:
up_write(&qm->qps_lock);

33
drivers/crypto/hisilicon/sec2/sec_crypto.c
View File

@ -118,7 +118,7 @@ struct sec_aead {
};
/* Get an en/de-cipher queue cyclically to balance load over queues of TFM */
static inline int sec_alloc_queue_id(struct sec_ctx *ctx, struct sec_req *req)
static inline u32 sec_alloc_queue_id(struct sec_ctx *ctx, struct sec_req *req)
{
if (req->c_req.encrypt)
return (u32)atomic_inc_return(&ctx->enc_qcyclic) %
@ -485,8 +485,7 @@ static void sec_alg_resource_free(struct sec_ctx *ctx,
sec_free_mac_resource(dev, qp_ctx->res);
}
static int sec_alloc_qp_ctx_resource(struct hisi_qm *qm, struct sec_ctx *ctx,
struct sec_qp_ctx *qp_ctx)
static int sec_alloc_qp_ctx_resource(struct sec_ctx *ctx, struct sec_qp_ctx *qp_ctx)
{
u16 q_depth = qp_ctx->qp->sq_depth;
struct device *dev = ctx->dev;
@ -541,8 +540,7 @@ static void sec_free_qp_ctx_resource(struct sec_ctx *ctx, struct sec_qp_ctx *qp_
kfree(qp_ctx->req_list);
}
static int sec_create_qp_ctx(struct hisi_qm *qm, struct sec_ctx *ctx,
int qp_ctx_id, int alg_type)
static int sec_create_qp_ctx(struct sec_ctx *ctx, int qp_ctx_id)
{
struct sec_qp_ctx *qp_ctx;
struct hisi_qp *qp;
@ -561,7 +559,7 @@ static int sec_create_qp_ctx(struct hisi_qm *qm, struct sec_ctx *ctx,
idr_init(&qp_ctx->req_idr);
INIT_LIST_HEAD(&qp_ctx->backlog);
ret = sec_alloc_qp_ctx_resource(qm, ctx, qp_ctx);
ret = sec_alloc_qp_ctx_resource(ctx, qp_ctx);
if (ret)
goto err_destroy_idr;
@ -614,7 +612,7 @@ static int sec_ctx_base_init(struct sec_ctx *ctx)
}
for (i = 0; i < sec->ctx_q_num; i++) {
ret = sec_create_qp_ctx(&sec->qm, ctx, i, 0);
ret = sec_create_qp_ctx(ctx, i);
if (ret)
goto err_sec_release_qp_ctx;
}
@ -750,9 +748,7 @@ static void sec_skcipher_uninit(struct crypto_skcipher *tfm)
sec_ctx_base_uninit(ctx);
}
static int sec_skcipher_3des_setkey(struct crypto_skcipher *tfm, const u8 *key,
const u32 keylen,
const enum sec_cmode c_mode)
static int sec_skcipher_3des_setkey(struct crypto_skcipher *tfm, const u8 *key, const u32 keylen)
{
struct sec_ctx *ctx = crypto_skcipher_ctx(tfm);
struct sec_cipher_ctx *c_ctx = &ctx->c_ctx;
@ -843,7 +839,7 @@ static int sec_skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key,
switch (c_alg) {
case SEC_CALG_3DES:
ret = sec_skcipher_3des_setkey(tfm, key, keylen, c_mode);
ret = sec_skcipher_3des_setkey(tfm, key, keylen);
break;
case SEC_CALG_AES:
case SEC_CALG_SM4:
@ -1371,7 +1367,7 @@ static int sec_skcipher_bd_fill_v3(struct sec_ctx *ctx, struct sec_req *req)
sec_sqe3->bd_param = cpu_to_le32(bd_param);
sec_sqe3->c_len_ivin |= cpu_to_le32(c_req->c_len);
sec_sqe3->tag = cpu_to_le64(req);
sec_sqe3->tag = cpu_to_le64((unsigned long)req);
return 0;
}
@ -2145,8 +2141,8 @@ static int sec_skcipher_decrypt(struct skcipher_request *sk_req)
return sec_skcipher_crypto(sk_req, false);
}
#define SEC_SKCIPHER_GEN_ALG(sec_cra_name, sec_set_key, sec_min_key_size, \
sec_max_key_size, ctx_init, ctx_exit, blk_size, iv_size)\
#define SEC_SKCIPHER_ALG(sec_cra_name, sec_set_key, \
sec_min_key_size, sec_max_key_size, blk_size, iv_size)\
{\
.base = {\
.cra_name = sec_cra_name,\
@ -2158,8 +2154,8 @@ static int sec_skcipher_decrypt(struct skcipher_request *sk_req)
.cra_ctxsize = sizeof(struct sec_ctx),\
.cra_module = THIS_MODULE,\
},\
.init = ctx_init,\
.exit = ctx_exit,\
.init = sec_skcipher_ctx_init,\
.exit = sec_skcipher_ctx_exit,\
.setkey = sec_set_key,\
.decrypt = sec_skcipher_decrypt,\
.encrypt = sec_skcipher_encrypt,\
@ -2168,11 +2164,6 @@ static int sec_skcipher_decrypt(struct skcipher_request *sk_req)
.ivsize = iv_size,\
}
#define SEC_SKCIPHER_ALG(name, key_func, min_key_size, \
max_key_size, blk_size, iv_size) \
SEC_SKCIPHER_GEN_ALG(name, key_func, min_key_size, max_key_size, \
sec_skcipher_ctx_init, sec_skcipher_ctx_exit, blk_size, iv_size)
static struct sec_skcipher sec_skciphers[] = {
{
.alg_msk = BIT(0),

7
drivers/crypto/hisilicon/sec2/sec_main.c
View File

@ -282,6 +282,11 @@ static const struct debugfs_reg32 sec_dfx_regs[] = {
{"SEC_BD_SAA6 ", 0x301C38},
{"SEC_BD_SAA7 ", 0x301C3C},
{"SEC_BD_SAA8 ", 0x301C40},
{"SEC_RAS_CE_ENABLE ", 0x301050},
{"SEC_RAS_FE_ENABLE ", 0x301054},
{"SEC_RAS_NFE_ENABLE ", 0x301058},
{"SEC_REQ_TRNG_TIME_TH ", 0x30112C},
{"SEC_CHANNEL_RNG_REQ_THLD ", 0x302110},
};
/* define the SEC's dfx regs region and region length */
@ -374,7 +379,7 @@ void sec_destroy_qps(struct hisi_qp **qps, int qp_num)
struct hisi_qp **sec_create_qps(void)
{
int node = cpu_to_node(smp_processor_id());
int node = cpu_to_node(raw_smp_processor_id());
u32 ctx_num = ctx_q_num;
struct hisi_qp **qps;
int ret;

1
drivers/crypto/hisilicon/zip/zip_crypto.c
View File

@ -591,6 +591,7 @@ static struct acomp_alg hisi_zip_acomp_deflate = {
.base = {
.cra_name = "deflate",
.cra_driver_name = "hisi-deflate-acomp",
.cra_flags = CRYPTO_ALG_ASYNC,
.cra_module = THIS_MODULE,
.cra_priority = HZIP_ALG_PRIORITY,
.cra_ctxsize = sizeof(struct hisi_zip_ctx),

2
drivers/crypto/hisilicon/zip/zip_main.c
View File

@ -454,7 +454,7 @@ MODULE_DEVICE_TABLE(pci, hisi_zip_dev_ids);
int zip_create_qps(struct hisi_qp **qps, int qp_num, int node)
{
if (node == NUMA_NO_NODE)
node = cpu_to_node(smp_processor_id());
node = cpu_to_node(raw_smp_processor_id());
return hisi_qm_alloc_qps_node(&zip_devices, qp_num, 0, node, qps);
}

25
drivers/crypto/intel/iaa/iaa_crypto.h
View File

@ -59,10 +59,8 @@ struct iaa_device_compression_mode {
const char *name;
struct aecs_comp_table_record *aecs_comp_table;
struct aecs_decomp_table_record *aecs_decomp_table;
dma_addr_t aecs_comp_table_dma_addr;
dma_addr_t aecs_decomp_table_dma_addr;
};
/* Representation of IAA device with wqs, populated by probe */
@ -107,23 +105,6 @@ struct aecs_comp_table_record {
u32 reserved_padding[2];
} __packed;
/* AECS for decompress */
struct aecs_decomp_table_record {
u32 crc;
u32 xor_checksum;
u32 low_filter_param;
u32 high_filter_param;
u32 output_mod_idx;
u32 drop_init_decomp_out_bytes;
u32 reserved[36];
u32 output_accum_data[2];
u32 out_bits_valid;
u32 bit_off_indexing;
u32 input_accum_data[64];
u8 size_qw[32];
u32 decomp_state[1220];
} __packed;
int iaa_aecs_init_fixed(void);
void iaa_aecs_cleanup_fixed(void);
@ -136,9 +117,6 @@ struct iaa_compression_mode {
int ll_table_size;
u32 *d_table;
int d_table_size;
u32 *header_table;
int header_table_size;
u16 gen_decomp_table_flags;
iaa_dev_comp_init_fn_t init;
iaa_dev_comp_free_fn_t free;
};
@ -148,9 +126,6 @@ int add_iaa_compression_mode(const char *name,
int ll_table_size,
const u32 *d_table,
int d_table_size,
const u8 *header_table,
int header_table_size,
u16 gen_decomp_table_flags,
iaa_dev_comp_init_fn_t init,
iaa_dev_comp_free_fn_t free);

1
drivers/crypto/intel/iaa/iaa_crypto_comp_fixed.c
View File

@ -78,7 +78,6 @@ int iaa_aecs_init_fixed(void)
sizeof(fixed_ll_sym),
fixed_d_sym,
sizeof(fixed_d_sym),
NULL, 0, 0,
init_fixed_mode, NULL);
if (!ret)
pr_debug("IAA fixed compression mode initialized\n");

122
drivers/crypto/intel/iaa/iaa_crypto_main.c
View File

@ -258,16 +258,14 @@ static void free_iaa_compression_mode(struct iaa_compression_mode *mode)
kfree(mode->name);
kfree(mode->ll_table);
kfree(mode->d_table);
kfree(mode->header_table);
kfree(mode);
}
/*
* IAA Compression modes are defined by an ll_table, a d_table, and an
* optional header_table. These tables are typically generated and
* captured using statistics collected from running actual
* compress/decompress workloads.
* IAA Compression modes are defined by an ll_table and a d_table.
* These tables are typically generated and captured using statistics
* collected from running actual compress/decompress workloads.
*
* A module or other kernel code can add and remove compression modes
* with a given name using the exported @add_iaa_compression_mode()
@ -315,9 +313,6 @@ EXPORT_SYMBOL_GPL(remove_iaa_compression_mode);
* @ll_table_size: The ll table size in bytes
* @d_table: The d table
* @d_table_size: The d table size in bytes
* @header_table: Optional header table
* @header_table_size: Optional header table size in bytes
* @gen_decomp_table_flags: Otional flags used to generate the decomp table
* @init: Optional callback function to init the compression mode data
* @free: Optional callback function to free the compression mode data
*
@ -330,9 +325,6 @@ int add_iaa_compression_mode(const char *name,
int ll_table_size,
const u32 *d_table,
int d_table_size,
const u8 *header_table,
int header_table_size,
u16 gen_decomp_table_flags,
iaa_dev_comp_init_fn_t init,
iaa_dev_comp_free_fn_t free)
{
@ -370,16 +362,6 @@ int add_iaa_compression_mode(const char *name,
mode->d_table_size = d_table_size;
}
if (header_table) {
mode->header_table = kzalloc(header_table_size, GFP_KERNEL);
if (!mode->header_table)
goto free;
memcpy(mode->header_table, header_table, header_table_size);
mode->header_table_size = header_table_size;
}
mode->gen_decomp_table_flags = gen_decomp_table_flags;
mode->init = init;
mode->free = free;
@ -420,10 +402,6 @@ static void free_device_compression_mode(struct iaa_device *iaa_device,
if (device_mode->aecs_comp_table)
dma_free_coherent(dev, size, device_mode->aecs_comp_table,
device_mode->aecs_comp_table_dma_addr);
if (device_mode->aecs_decomp_table)
dma_free_coherent(dev, size, device_mode->aecs_decomp_table,
device_mode->aecs_decomp_table_dma_addr);
kfree(device_mode);
}
@ -440,73 +418,6 @@ static int check_completion(struct device *dev,
bool compress,
bool only_once);
static int decompress_header(struct iaa_device_compression_mode *device_mode,
struct iaa_compression_mode *mode,
struct idxd_wq *wq)
{
dma_addr_t src_addr, src2_addr;
struct idxd_desc *idxd_desc;
struct iax_hw_desc *desc;
struct device *dev;
int ret = 0;
idxd_desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK);
if (IS_ERR(idxd_desc))
return PTR_ERR(idxd_desc);
desc = idxd_desc->iax_hw;
dev = &wq->idxd->pdev->dev;
src_addr = dma_map_single(dev, (void *)mode->header_table,
mode->header_table_size, DMA_TO_DEVICE);
dev_dbg(dev, "%s: mode->name %s, src_addr %llx, dev %p, src %p, slen %d\n",
__func__, mode->name, src_addr, dev,
mode->header_table, mode->header_table_size);
if (unlikely(dma_mapping_error(dev, src_addr))) {
dev_dbg(dev, "dma_map_single err, exiting\n");
ret = -ENOMEM;
return ret;
}
desc->flags = IAX_AECS_GEN_FLAG;
desc->opcode = IAX_OPCODE_DECOMPRESS;
desc->src1_addr = (u64)src_addr;
desc->src1_size = mode->header_table_size;
src2_addr = device_mode->aecs_decomp_table_dma_addr;
desc->src2_addr = (u64)src2_addr;
desc->src2_size = 1088;
dev_dbg(dev, "%s: mode->name %s, src2_addr %llx, dev %p, src2_size %d\n",
__func__, mode->name, desc->src2_addr, dev, desc->src2_size);
desc->max_dst_size = 0; // suppressed output
desc->decompr_flags = mode->gen_decomp_table_flags;
desc->priv = 0;
desc->completion_addr = idxd_desc->compl_dma;
ret = idxd_submit_desc(wq, idxd_desc);
if (ret) {
pr_err("%s: submit_desc failed ret=0x%x\n", __func__, ret);
goto out;
}
ret = check_completion(dev, idxd_desc->iax_completion, false, false);
if (ret)
dev_dbg(dev, "%s: mode->name %s check_completion failed ret=%d\n",
__func__, mode->name, ret);
else
dev_dbg(dev, "%s: mode->name %s succeeded\n", __func__,
mode->name);
out:
dma_unmap_single(dev, src_addr, 1088, DMA_TO_DEVICE);
return ret;
}
static int init_device_compression_mode(struct iaa_device *iaa_device,
struct iaa_compression_mode *mode,
int idx, struct idxd_wq *wq)
@ -529,24 +440,11 @@ static int init_device_compression_mode(struct iaa_device *iaa_device,
if (!device_mode->aecs_comp_table)
goto free;
device_mode->aecs_decomp_table = dma_alloc_coherent(dev, size,
&device_mode->aecs_decomp_table_dma_addr, GFP_KERNEL);
if (!device_mode->aecs_decomp_table)
goto free;
/* Add Huffman table to aecs */
memset(device_mode->aecs_comp_table, 0, sizeof(*device_mode->aecs_comp_table));
memcpy(device_mode->aecs_comp_table->ll_sym, mode->ll_table, mode->ll_table_size);
memcpy(device_mode->aecs_comp_table->d_sym, mode->d_table, mode->d_table_size);
if (mode->header_table) {
ret = decompress_header(device_mode, mode, wq);
if (ret) {
pr_debug("iaa header decompression failed: ret=%d\n", ret);
goto free;
}
}
if (mode->init) {
ret = mode->init(device_mode);
if (ret)
@ -1324,7 +1222,7 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req,
*compression_crc = idxd_desc->iax_completion->crc;
if (!ctx->async_mode)
if (!ctx->async_mode || disable_async)
idxd_free_desc(wq, idxd_desc);
out:
return ret;
@ -1570,7 +1468,7 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req,
*dlen = req->dlen;
if (!ctx->async_mode)
if (!ctx->async_mode || disable_async)
idxd_free_desc(wq, idxd_desc);
/* Update stats */
@ -1596,6 +1494,7 @@ static int iaa_comp_acompress(struct acomp_req *req)
u32 compression_crc;
struct idxd_wq *wq;
struct device *dev;
u64 start_time_ns;
int order = -1;
compression_ctx = crypto_tfm_ctx(tfm);
@ -1669,8 +1568,10 @@ static int iaa_comp_acompress(struct acomp_req *req)
" req->dlen %d, sg_dma_len(sg) %d\n", dst_addr, nr_sgs,
req->dst, req->dlen, sg_dma_len(req->dst));
start_time_ns = iaa_get_ts();
ret = iaa_compress(tfm, req, wq, src_addr, req->slen, dst_addr,
&req->dlen, &compression_crc, disable_async);
update_max_comp_delay_ns(start_time_ns);
if (ret == -EINPROGRESS)
return ret;
@ -1717,6 +1618,7 @@ static int iaa_comp_adecompress_alloc_dest(struct acomp_req *req)
struct iaa_wq *iaa_wq;
struct device *dev;
struct idxd_wq *wq;
u64 start_time_ns;
int order = -1;
cpu = get_cpu();
@ -1773,8 +1675,10 @@ alloc_dest:
dev_dbg(dev, "dma_map_sg, dst_addr %llx, nr_sgs %d, req->dst %p,"
" req->dlen %d, sg_dma_len(sg) %d\n", dst_addr, nr_sgs,
req->dst, req->dlen, sg_dma_len(req->dst));
start_time_ns = iaa_get_ts();
ret = iaa_decompress(tfm, req, wq, src_addr, req->slen,
dst_addr, &req->dlen, true);
update_max_decomp_delay_ns(start_time_ns);
if (ret == -EOVERFLOW) {
dma_unmap_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE);
req->dlen *= 2;
@ -1805,6 +1709,7 @@ static int iaa_comp_adecompress(struct acomp_req *req)
int nr_sgs, cpu, ret = 0;
struct iaa_wq *iaa_wq;
struct device *dev;
u64 start_time_ns;
struct idxd_wq *wq;
if (!iaa_crypto_enabled) {
@ -1864,8 +1769,10 @@ static int iaa_comp_adecompress(struct acomp_req *req)
" req->dlen %d, sg_dma_len(sg) %d\n", dst_addr, nr_sgs,
req->dst, req->dlen, sg_dma_len(req->dst));
start_time_ns = iaa_get_ts();
ret = iaa_decompress(tfm, req, wq, src_addr, req->slen,
dst_addr, &req->dlen, false);
update_max_decomp_delay_ns(start_time_ns);
if (ret == -EINPROGRESS)
return ret;
@ -1916,6 +1823,7 @@ static struct acomp_alg iaa_acomp_fixed_deflate = {
.base = {
.cra_name = "deflate",
.cra_driver_name = "deflate-iaa",
.cra_flags = CRYPTO_ALG_ASYNC,
.cra_ctxsize = sizeof(struct iaa_compression_ctx),
.cra_module = THIS_MODULE,
.cra_priority = IAA_ALG_PRIORITY,

30
drivers/crypto/intel/iaa/iaa_crypto_stats.c
View File

@ -22,8 +22,6 @@ static u64 total_decomp_calls;
static u64 total_sw_decomp_calls;
static u64 max_comp_delay_ns;
static u64 max_decomp_delay_ns;
static u64 max_acomp_delay_ns;
static u64 max_adecomp_delay_ns;
static u64 total_comp_bytes_out;
static u64 total_decomp_bytes_in;
static u64 total_completion_einval_errors;
@ -92,26 +90,6 @@ void update_max_decomp_delay_ns(u64 start_time_ns)
max_decomp_delay_ns = time_diff;
}
void update_max_acomp_delay_ns(u64 start_time_ns)
{
u64 time_diff;
time_diff = ktime_get_ns() - start_time_ns;
if (time_diff > max_acomp_delay_ns)
max_acomp_delay_ns = time_diff;
}
void update_max_adecomp_delay_ns(u64 start_time_ns)
{
u64 time_diff;
time_diff = ktime_get_ns() - start_time_ns;
if (time_diff > max_adecomp_delay_ns)
max_adecomp_delay_ns = time_diff;
}
void update_wq_comp_calls(struct idxd_wq *idxd_wq)
{
struct iaa_wq *wq = idxd_wq_get_private(idxd_wq);
@ -151,8 +129,6 @@ static void reset_iaa_crypto_stats(void)
total_sw_decomp_calls = 0;
max_comp_delay_ns = 0;
max_decomp_delay_ns = 0;
max_acomp_delay_ns = 0;
max_adecomp_delay_ns = 0;
total_comp_bytes_out = 0;
total_decomp_bytes_in = 0;
total_completion_einval_errors = 0;
@ -275,17 +251,11 @@ int __init iaa_crypto_debugfs_init(void)
return -ENODEV;
iaa_crypto_debugfs_root = debugfs_create_dir("iaa_crypto", NULL);
if (!iaa_crypto_debugfs_root)
return -ENOMEM;
debugfs_create_u64("max_comp_delay_ns", 0644,
iaa_crypto_debugfs_root, &max_comp_delay_ns);
debugfs_create_u64("max_decomp_delay_ns", 0644,
iaa_crypto_debugfs_root, &max_decomp_delay_ns);
debugfs_create_u64("max_acomp_delay_ns", 0644,
iaa_crypto_debugfs_root, &max_comp_delay_ns);
debugfs_create_u64("max_adecomp_delay_ns", 0644,
iaa_crypto_debugfs_root, &max_decomp_delay_ns);
debugfs_create_u64("total_comp_calls", 0644,
iaa_crypto_debugfs_root, &total_comp_calls);
debugfs_create_u64("total_decomp_calls", 0644,

8
drivers/crypto/intel/iaa/iaa_crypto_stats.h
View File

@ -15,8 +15,6 @@ void update_total_sw_decomp_calls(void);
void update_total_decomp_bytes_in(int n);
void update_max_comp_delay_ns(u64 start_time_ns);
void update_max_decomp_delay_ns(u64 start_time_ns);
void update_max_acomp_delay_ns(u64 start_time_ns);
void update_max_adecomp_delay_ns(u64 start_time_ns);
void update_completion_einval_errs(void);
void update_completion_timeout_errs(void);
void update_completion_comp_buf_overflow_errs(void);
@ -26,6 +24,8 @@ void update_wq_comp_bytes(struct idxd_wq *idxd_wq, int n);
void update_wq_decomp_calls(struct idxd_wq *idxd_wq);
void update_wq_decomp_bytes(struct idxd_wq *idxd_wq, int n);
static inline u64 iaa_get_ts(void) { return ktime_get_ns(); }
#else
static inline int iaa_crypto_debugfs_init(void) { return 0; }
static inline void iaa_crypto_debugfs_cleanup(void) {}
@ -37,8 +37,6 @@ static inline void update_total_sw_decomp_calls(void) {}
static inline void update_total_decomp_bytes_in(int n) {}
static inline void update_max_comp_delay_ns(u64 start_time_ns) {}
static inline void update_max_decomp_delay_ns(u64 start_time_ns) {}
static inline void update_max_acomp_delay_ns(u64 start_time_ns) {}
static inline void update_max_adecomp_delay_ns(u64 start_time_ns) {}
static inline void update_completion_einval_errs(void) {}
static inline void update_completion_timeout_errs(void) {}
static inline void update_completion_comp_buf_overflow_errs(void) {}
@ -48,6 +46,8 @@ static inline void update_wq_comp_bytes(struct idxd_wq *idxd_wq, int n) {}
static inline void update_wq_decomp_calls(struct idxd_wq *idxd_wq) {}
static inline void update_wq_decomp_bytes(struct idxd_wq *idxd_wq, int n) {}
static inline u64 iaa_get_ts(void) { return 0; }
#endif // CONFIG_CRYPTO_DEV_IAA_CRYPTO_STATS
#endif

14
drivers/crypto/intel/qat/Kconfig
View File

@ -106,3 +106,17 @@ config CRYPTO_DEV_QAT_C62XVF
To compile this as a module, choose M here: the module
will be called qat_c62xvf.
config CRYPTO_DEV_QAT_ERROR_INJECTION
bool "Support for Intel(R) QAT Devices Heartbeat Error Injection"
depends on CRYPTO_DEV_QAT
depends on DEBUG_FS
help
Enables a mechanism that allows to inject a heartbeat error on
Intel(R) QuickAssist devices for testing purposes.
This is intended for developer use only.
If unsure, say N.
This functionality is available via debugfs entry of the Intel(R)
QuickAssist device

64
drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c
View File

@ -361,53 +361,6 @@ static u32 get_ena_thd_mask(struct adf_accel_dev *accel_dev, u32 obj_num)
}
}
static u16 get_ring_to_svc_map(struct adf_accel_dev *accel_dev)
{
enum adf_cfg_service_type rps[RP_GROUP_COUNT] = { };
const struct adf_fw_config *fw_config;
u16 ring_to_svc_map;
int i, j;
fw_config = get_fw_config(accel_dev);
if (!fw_config)
return 0;
for (i = 0; i < RP_GROUP_COUNT; i++) {
switch (fw_config[i].ae_mask) {
case ADF_AE_GROUP_0:
j = RP_GROUP_0;
break;
case ADF_AE_GROUP_1:
j = RP_GROUP_1;
break;
default:
return 0;
}
switch (fw_config[i].obj) {
case ADF_FW_SYM_OBJ:
rps[j] = SYM;
break;
case ADF_FW_ASYM_OBJ:
rps[j] = ASYM;
break;
case ADF_FW_DC_OBJ:
rps[j] = COMP;
break;
default:
rps[j] = 0;
break;
}
}
ring_to_svc_map = rps[RP_GROUP_0] << ADF_CFG_SERV_RING_PAIR_0_SHIFT |
rps[RP_GROUP_1] << ADF_CFG_SERV_RING_PAIR_1_SHIFT |
rps[RP_GROUP_0] << ADF_CFG_SERV_RING_PAIR_2_SHIFT |
rps[RP_GROUP_1] << ADF_CFG_SERV_RING_PAIR_3_SHIFT;
return ring_to_svc_map;
}
static const char *uof_get_name(struct adf_accel_dev *accel_dev, u32 obj_num,
const char * const fw_objs[], int num_objs)
{
@ -433,6 +386,20 @@ static const char *uof_get_name_420xx(struct adf_accel_dev *accel_dev, u32 obj_n
return uof_get_name(accel_dev, obj_num, adf_420xx_fw_objs, num_fw_objs);
}
static int uof_get_obj_type(struct adf_accel_dev *accel_dev, u32 obj_num)
{
const struct adf_fw_config *fw_config;
if (obj_num >= uof_get_num_objs(accel_dev))
return -EINVAL;
fw_config = get_fw_config(accel_dev);
if (!fw_config)
return -EINVAL;
return fw_config[obj_num].obj;
}
static u32 uof_get_ae_mask(struct adf_accel_dev *accel_dev, u32 obj_num)
{
const struct adf_fw_config *fw_config;
@ -496,12 +463,13 @@ void adf_init_hw_data_420xx(struct adf_hw_device_data *hw_data, u32 dev_id)
hw_data->fw_mmp_name = ADF_420XX_MMP;
hw_data->uof_get_name = uof_get_name_420xx;
hw_data->uof_get_num_objs = uof_get_num_objs;
hw_data->uof_get_obj_type = uof_get_obj_type;
hw_data->uof_get_ae_mask = uof_get_ae_mask;
hw_data->get_rp_group = get_rp_group;
hw_data->get_ena_thd_mask = get_ena_thd_mask;
hw_data->set_msix_rttable = adf_gen4_set_msix_default_rttable;
hw_data->set_ssm_wdtimer = adf_gen4_set_ssm_wdtimer;
hw_data->get_ring_to_svc_map = get_ring_to_svc_map;
hw_data->get_ring_to_svc_map = adf_gen4_get_ring_to_svc_map;
hw_data->disable_iov = adf_disable_sriov;
hw_data->ring_pair_reset = adf_gen4_ring_pair_reset;
hw_data->enable_pm = adf_gen4_enable_pm;

64
drivers/crypto/intel/qat/qat_4xxx/adf_4xxx_hw_data.c
View File

@ -320,53 +320,6 @@ static u32 get_ena_thd_mask_401xx(struct adf_accel_dev *accel_dev, u32 obj_num)
}
}
static u16 get_ring_to_svc_map(struct adf_accel_dev *accel_dev)
{
enum adf_cfg_service_type rps[RP_GROUP_COUNT];
const struct adf_fw_config *fw_config;
u16 ring_to_svc_map;
int i, j;
fw_config = get_fw_config(accel_dev);
if (!fw_config)
return 0;
for (i = 0; i < RP_GROUP_COUNT; i++) {
switch (fw_config[i].ae_mask) {
case ADF_AE_GROUP_0:
j = RP_GROUP_0;
break;
case ADF_AE_GROUP_1:
j = RP_GROUP_1;
break;
default:
return 0;
}
switch (fw_config[i].obj) {
case ADF_FW_SYM_OBJ:
rps[j] = SYM;
break;
case ADF_FW_ASYM_OBJ:
rps[j] = ASYM;
break;
case ADF_FW_DC_OBJ:
rps[j] = COMP;
break;
default:
rps[j] = 0;
break;
}
}
ring_to_svc_map = rps[RP_GROUP_0] << ADF_CFG_SERV_RING_PAIR_0_SHIFT |
rps[RP_GROUP_1] << ADF_CFG_SERV_RING_PAIR_1_SHIFT |
rps[RP_GROUP_0] << ADF_CFG_SERV_RING_PAIR_2_SHIFT |
rps[RP_GROUP_1] << ADF_CFG_SERV_RING_PAIR_3_SHIFT;
return ring_to_svc_map;
}
static const char *uof_get_name(struct adf_accel_dev *accel_dev, u32 obj_num,
const char * const fw_objs[], int num_objs)
{
@ -399,6 +352,20 @@ static const char *uof_get_name_402xx(struct adf_accel_dev *accel_dev, u32 obj_n
return uof_get_name(accel_dev, obj_num, adf_402xx_fw_objs, num_fw_objs);
}
static int uof_get_obj_type(struct adf_accel_dev *accel_dev, u32 obj_num)
{
const struct adf_fw_config *fw_config;
if (obj_num >= uof_get_num_objs(accel_dev))
return -EINVAL;
fw_config = get_fw_config(accel_dev);
if (!fw_config)
return -EINVAL;
return fw_config[obj_num].obj;
}
static u32 uof_get_ae_mask(struct adf_accel_dev *accel_dev, u32 obj_num)
{
const struct adf_fw_config *fw_config;
@ -479,11 +446,12 @@ void adf_init_hw_data_4xxx(struct adf_hw_device_data *hw_data, u32 dev_id)
break;
}
hw_data->uof_get_num_objs = uof_get_num_objs;
hw_data->uof_get_obj_type = uof_get_obj_type;
hw_data->uof_get_ae_mask = uof_get_ae_mask;
hw_data->get_rp_group = get_rp_group;
hw_data->set_msix_rttable = adf_gen4_set_msix_default_rttable;
hw_data->set_ssm_wdtimer = adf_gen4_set_ssm_wdtimer;
hw_data->get_ring_to_svc_map = get_ring_to_svc_map;
hw_data->get_ring_to_svc_map = adf_gen4_get_ring_to_svc_map;
hw_data->disable_iov = adf_disable_sriov;
hw_data->ring_pair_reset = adf_gen4_ring_pair_reset;
hw_data->enable_pm = adf_gen4_enable_pm;

2
drivers/crypto/intel/qat/qat_common/Makefile
View File

@ -53,3 +53,5 @@ intel_qat-$(CONFIG_PCI_IOV) += adf_sriov.o adf_vf_isr.o adf_pfvf_utils.o \
adf_pfvf_pf_msg.o adf_pfvf_pf_proto.o \
adf_pfvf_vf_msg.o adf_pfvf_vf_proto.o \
adf_gen2_pfvf.o adf_gen4_pfvf.o
intel_qat-$(CONFIG_CRYPTO_DEV_QAT_ERROR_INJECTION) += adf_heartbeat_inject.o

3
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h
View File

@ -248,6 +248,7 @@ struct adf_hw_device_data {
void (*set_msix_rttable)(struct adf_accel_dev *accel_dev);
const char *(*uof_get_name)(struct adf_accel_dev *accel_dev, u32 obj_num);
u32 (*uof_get_num_objs)(struct adf_accel_dev *accel_dev);
int (*uof_get_obj_type)(struct adf_accel_dev *accel_dev, u32 obj_num);
u32 (*uof_get_ae_mask)(struct adf_accel_dev *accel_dev, u32 obj_num);
int (*get_rp_group)(struct adf_accel_dev *accel_dev, u32 ae_mask);
u32 (*get_ena_thd_mask)(struct adf_accel_dev *accel_dev, u32 obj_num);
@ -332,6 +333,7 @@ struct adf_accel_vf_info {
struct ratelimit_state vf2pf_ratelimit;
u32 vf_nr;
bool init;
bool restarting;
u8 vf_compat_ver;
};
@ -401,6 +403,7 @@ struct adf_accel_dev {
struct adf_error_counters ras_errors;
struct mutex state_lock; /* protect state of the device */
bool is_vf;
bool autoreset_on_error;
u32 accel_id;
};
#endif

138
drivers/crypto/intel/qat/qat_common/adf_aer.c
View File

@ -7,8 +7,15 @@
#include <linux/delay.h>
#include "adf_accel_devices.h"
#include "adf_common_drv.h"
#include "adf_pfvf_pf_msg.h"
struct adf_fatal_error_data {
struct adf_accel_dev *accel_dev;
struct work_struct work;
};
static struct workqueue_struct *device_reset_wq;
static struct workqueue_struct *device_sriov_wq;
static pci_ers_result_t adf_error_detected(struct pci_dev *pdev,
pci_channel_state_t state)
@ -26,6 +33,19 @@ static pci_ers_result_t adf_error_detected(struct pci_dev *pdev,
return PCI_ERS_RESULT_DISCONNECT;
}
set_bit(ADF_STATUS_RESTARTING, &accel_dev->status);
if (accel_dev->hw_device->exit_arb) {
dev_dbg(&pdev->dev, "Disabling arbitration\n");
accel_dev->hw_device->exit_arb(accel_dev);
}
adf_error_notifier(accel_dev);
adf_pf2vf_notify_fatal_error(accel_dev);
adf_dev_restarting_notify(accel_dev);
adf_pf2vf_notify_restarting(accel_dev);
adf_pf2vf_wait_for_restarting_complete(accel_dev);
pci_clear_master(pdev);
adf_dev_down(accel_dev, false);
return PCI_ERS_RESULT_NEED_RESET;
}
@ -37,6 +57,13 @@ struct adf_reset_dev_data {
struct work_struct reset_work;
};
/* sriov dev data */
struct adf_sriov_dev_data {
struct adf_accel_dev *accel_dev;
struct completion compl;
struct work_struct sriov_work;
};
void adf_reset_sbr(struct adf_accel_dev *accel_dev)
{
struct pci_dev *pdev = accel_to_pci_dev(accel_dev);
@ -82,29 +109,57 @@ void adf_dev_restore(struct adf_accel_dev *accel_dev)
}
}
static void adf_device_sriov_worker(struct work_struct *work)
{
struct adf_sriov_dev_data *sriov_data =
container_of(work, struct adf_sriov_dev_data, sriov_work);
adf_reenable_sriov(sriov_data->accel_dev);
complete(&sriov_data->compl);
}
static void adf_device_reset_worker(struct work_struct *work)
{
struct adf_reset_dev_data *reset_data =
container_of(work, struct adf_reset_dev_data, reset_work);
struct adf_accel_dev *accel_dev = reset_data->accel_dev;
unsigned long wait_jiffies = msecs_to_jiffies(10000);
struct adf_sriov_dev_data sriov_data;
adf_dev_restarting_notify(accel_dev);
if (adf_dev_restart(accel_dev)) {
/* The device hanged and we can't restart it so stop here */
dev_err(&GET_DEV(accel_dev), "Restart device failed\n");
if (reset_data->mode == ADF_DEV_RESET_ASYNC)
if (reset_data->mode == ADF_DEV_RESET_ASYNC ||
completion_done(&reset_data->compl))
kfree(reset_data);
WARN(1, "QAT: device restart failed. Device is unusable\n");
return;
}
sriov_data.accel_dev = accel_dev;
init_completion(&sriov_data.compl);
INIT_WORK(&sriov_data.sriov_work, adf_device_sriov_worker);
queue_work(device_sriov_wq, &sriov_data.sriov_work);
if (wait_for_completion_timeout(&sriov_data.compl, wait_jiffies))
adf_pf2vf_notify_restarted(accel_dev);
adf_dev_restarted_notify(accel_dev);
clear_bit(ADF_STATUS_RESTARTING, &accel_dev->status);
/* The dev is back alive. Notify the caller if in sync mode */
if (reset_data->mode == ADF_DEV_RESET_SYNC)
complete(&reset_data->compl);
else
/*
* The dev is back alive. Notify the caller if in sync mode
*
* If device restart will take a more time than expected,
* the schedule_reset() function can timeout and exit. This can be
* detected by calling the completion_done() function. In this case
* the reset_data structure needs to be freed here.
*/
if (reset_data->mode == ADF_DEV_RESET_ASYNC ||
completion_done(&reset_data->compl))
kfree(reset_data);
else
complete(&reset_data->compl);
}
static int adf_dev_aer_schedule_reset(struct adf_accel_dev *accel_dev,
@ -137,8 +192,9 @@ static int adf_dev_aer_schedule_reset(struct adf_accel_dev *accel_dev,
dev_err(&GET_DEV(accel_dev),
"Reset device timeout expired\n");
ret = -EFAULT;
} else {
kfree(reset_data);
}
kfree(reset_data);
return ret;
}
return 0;
@ -147,14 +203,25 @@ static int adf_dev_aer_schedule_reset(struct adf_accel_dev *accel_dev,
static pci_ers_result_t adf_slot_reset(struct pci_dev *pdev)
{
struct adf_accel_dev *accel_dev = adf_devmgr_pci_to_accel_dev(pdev);
int res = 0;
if (!accel_dev) {
pr_err("QAT: Can't find acceleration device\n");
return PCI_ERS_RESULT_DISCONNECT;
}
if (adf_dev_aer_schedule_reset(accel_dev, ADF_DEV_RESET_SYNC))
if (!pdev->is_busmaster)
pci_set_master(pdev);
pci_restore_state(pdev);
pci_save_state(pdev);
res = adf_dev_up(accel_dev, false);
if (res && res != -EALREADY)
return PCI_ERS_RESULT_DISCONNECT;
adf_reenable_sriov(accel_dev);
adf_pf2vf_notify_restarted(accel_dev);
adf_dev_restarted_notify(accel_dev);
clear_bit(ADF_STATUS_RESTARTING, &accel_dev->status);
return PCI_ERS_RESULT_RECOVERED;
}
@ -171,11 +238,62 @@ const struct pci_error_handlers adf_err_handler = {
};
EXPORT_SYMBOL_GPL(adf_err_handler);
int adf_dev_autoreset(struct adf_accel_dev *accel_dev)
{
if (accel_dev->autoreset_on_error)
return adf_dev_aer_schedule_reset(accel_dev, ADF_DEV_RESET_ASYNC);
return 0;
}
static void adf_notify_fatal_error_worker(struct work_struct *work)
{
struct adf_fatal_error_data *wq_data =
container_of(work, struct adf_fatal_error_data, work);
struct adf_accel_dev *accel_dev = wq_data->accel_dev;
struct adf_hw_device_data *hw_device = accel_dev->hw_device;
adf_error_notifier(accel_dev);
if (!accel_dev->is_vf) {
/* Disable arbitration to stop processing of new requests */
if (accel_dev->autoreset_on_error && hw_device->exit_arb)
hw_device->exit_arb(accel_dev);
if (accel_dev->pf.vf_info)
adf_pf2vf_notify_fatal_error(accel_dev);
adf_dev_autoreset(accel_dev);
}
kfree(wq_data);
}
int adf_notify_fatal_error(struct adf_accel_dev *accel_dev)
{
struct adf_fatal_error_data *wq_data;
wq_data = kzalloc(sizeof(*wq_data), GFP_ATOMIC);
if (!wq_data)
return -ENOMEM;
wq_data->accel_dev = accel_dev;
INIT_WORK(&wq_data->work, adf_notify_fatal_error_worker);
adf_misc_wq_queue_work(&wq_data->work);
return 0;
}
int adf_init_aer(void)
{
device_reset_wq = alloc_workqueue("qat_device_reset_wq",
WQ_MEM_RECLAIM, 0);
return !device_reset_wq ? -EFAULT : 0;
if (!device_reset_wq)
return -EFAULT;
device_sriov_wq = alloc_workqueue("qat_device_sriov_wq", 0, 0);
if (!device_sriov_wq)
return -EFAULT;
return 0;
}
void adf_exit_aer(void)
@ -183,4 +301,8 @@ void adf_exit_aer(void)
if (device_reset_wq)
destroy_workqueue(device_reset_wq);
device_reset_wq = NULL;
if (device_sriov_wq)
destroy_workqueue(device_sriov_wq);
device_sriov_wq = NULL;
}

1
drivers/crypto/intel/qat/qat_common/adf_cfg_strings.h
View File

@ -49,5 +49,6 @@
ADF_ETRMGR_BANK "%d" ADF_ETRMGR_CORE_AFFINITY
#define ADF_ACCEL_STR "Accelerator%d"
#define ADF_HEARTBEAT_TIMER "HeartbeatTimer"
#define ADF_SRIOV_ENABLED "SriovEnabled"
#endif

3
drivers/crypto/intel/qat/qat_common/adf_clock.c
View File

@ -83,6 +83,9 @@ static int measure_clock(struct adf_accel_dev *accel_dev, u32 *frequency)
}
delta_us = timespec_to_us(&ts3) - timespec_to_us(&ts1);
if (!delta_us)
return -EINVAL;
temp = (timestamp2 - timestamp1) * ME_CLK_DIVIDER * 10;
temp = DIV_ROUND_CLOSEST_ULL(temp, delta_us);
/*

1
drivers/crypto/intel/qat/qat_common/adf_cnv_dbgfs.c
View File

@ -16,7 +16,6 @@
#define CNV_ERR_INFO_MASK GENMASK(11, 0)
#define CNV_ERR_TYPE_MASK GENMASK(15, 12)
#define CNV_SLICE_ERR_MASK GENMASK(7, 0)
#define CNV_SLICE_ERR_SIGN_BIT_INDEX 7
#define CNV_DELTA_ERR_SIGN_BIT_INDEX 11

10
drivers/crypto/intel/qat/qat_common/adf_common_drv.h
View File

@ -40,6 +40,7 @@ enum adf_event {
ADF_EVENT_SHUTDOWN,
ADF_EVENT_RESTARTING,
ADF_EVENT_RESTARTED,
ADF_EVENT_FATAL_ERROR,
};
struct service_hndl {
@ -60,6 +61,8 @@ int adf_dev_restart(struct adf_accel_dev *accel_dev);
void adf_devmgr_update_class_index(struct adf_hw_device_data *hw_data);
void adf_clean_vf_map(bool);
int adf_notify_fatal_error(struct adf_accel_dev *accel_dev);
void adf_error_notifier(struct adf_accel_dev *accel_dev);
int adf_devmgr_add_dev(struct adf_accel_dev *accel_dev,
struct adf_accel_dev *pf);
void adf_devmgr_rm_dev(struct adf_accel_dev *accel_dev,
@ -84,12 +87,14 @@ int adf_ae_stop(struct adf_accel_dev *accel_dev);
extern const struct pci_error_handlers adf_err_handler;
void adf_reset_sbr(struct adf_accel_dev *accel_dev);
void adf_reset_flr(struct adf_accel_dev *accel_dev);
int adf_dev_autoreset(struct adf_accel_dev *accel_dev);
void adf_dev_restore(struct adf_accel_dev *accel_dev);
int adf_init_aer(void);
void adf_exit_aer(void);
int adf_init_arb(struct adf_accel_dev *accel_dev);
void adf_exit_arb(struct adf_accel_dev *accel_dev);
void adf_update_ring_arb(struct adf_etr_ring_data *ring);
int adf_disable_arb_thd(struct adf_accel_dev *accel_dev, u32 ae, u32 thr);
int adf_dev_get(struct adf_accel_dev *accel_dev);
void adf_dev_put(struct adf_accel_dev *accel_dev);
@ -188,6 +193,7 @@ bool adf_misc_wq_queue_delayed_work(struct delayed_work *work,
#if defined(CONFIG_PCI_IOV)
int adf_sriov_configure(struct pci_dev *pdev, int numvfs);
void adf_disable_sriov(struct adf_accel_dev *accel_dev);
void adf_reenable_sriov(struct adf_accel_dev *accel_dev);
void adf_enable_vf2pf_interrupts(struct adf_accel_dev *accel_dev, u32 vf_mask);
void adf_disable_all_vf2pf_interrupts(struct adf_accel_dev *accel_dev);
bool adf_recv_and_handle_pf2vf_msg(struct adf_accel_dev *accel_dev);
@ -208,6 +214,10 @@ static inline void adf_disable_sriov(struct adf_accel_dev *accel_dev)
{
}
static inline void adf_reenable_sriov(struct adf_accel_dev *accel_dev)
{
}
static inline int adf_init_pf_wq(void)
{
return 0;

4
drivers/crypto/intel/qat/qat_common/adf_dev_mgr.c
View File

@ -60,10 +60,10 @@ static int adf_get_vf_real_id(u32 fake)
/**
* adf_clean_vf_map() - Cleans VF id mapings
*
* Function cleans internal ids for virtual functions.
* @vf: flag indicating whether mappings is cleaned
* for vfs only or for vfs and pfs
*
* Function cleans internal ids for virtual functions.
*/
void adf_clean_vf_map(bool vf)
{

59
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c
View File

@ -4,6 +4,7 @@
#include "adf_accel_devices.h"
#include "adf_cfg_services.h"
#include "adf_common_drv.h"
#include "adf_fw_config.h"
#include "adf_gen4_hw_data.h"
#include "adf_gen4_pm.h"
@ -398,6 +399,9 @@ int adf_gen4_init_thd2arb_map(struct adf_accel_dev *accel_dev)
ADF_GEN4_ADMIN_ACCELENGINES;
if (srv_id == SVC_DCC) {
if (ae_cnt > ICP_QAT_HW_AE_DELIMITER)
return -EINVAL;
memcpy(thd2arb_map, thrd_to_arb_map_dcc,
array_size(sizeof(*thd2arb_map), ae_cnt));
return 0;
@ -430,3 +434,58 @@ int adf_gen4_init_thd2arb_map(struct adf_accel_dev *accel_dev)
return 0;
}
EXPORT_SYMBOL_GPL(adf_gen4_init_thd2arb_map);
u16 adf_gen4_get_ring_to_svc_map(struct adf_accel_dev *accel_dev)
{
struct adf_hw_device_data *hw_data = GET_HW_DATA(accel_dev);
enum adf_cfg_service_type rps[RP_GROUP_COUNT] = { };
unsigned int ae_mask, start_id, worker_obj_cnt, i;
u16 ring_to_svc_map;
int rp_group;
if (!hw_data->get_rp_group || !hw_data->uof_get_ae_mask ||
!hw_data->uof_get_obj_type || !hw_data->uof_get_num_objs)
return 0;
/* If dcc, all rings handle compression requests */
if (adf_get_service_enabled(accel_dev) == SVC_DCC) {
for (i = 0; i < RP_GROUP_COUNT; i++)
rps[i] = COMP;
goto set_mask;
}
worker_obj_cnt = hw_data->uof_get_num_objs(accel_dev) -
ADF_GEN4_ADMIN_ACCELENGINES;
start_id = worker_obj_cnt - RP_GROUP_COUNT;
for (i = start_id; i < worker_obj_cnt; i++) {
ae_mask = hw_data->uof_get_ae_mask(accel_dev, i);
rp_group = hw_data->get_rp_group(accel_dev, ae_mask);
if (rp_group >= RP_GROUP_COUNT || rp_group < RP_GROUP_0)
return 0;
switch (hw_data->uof_get_obj_type(accel_dev, i)) {
case ADF_FW_SYM_OBJ:
rps[rp_group] = SYM;
break;
case ADF_FW_ASYM_OBJ:
rps[rp_group] = ASYM;
break;
case ADF_FW_DC_OBJ:
rps[rp_group] = COMP;
break;
default:
rps[rp_group] = 0;
break;
}
}
set_mask:
ring_to_svc_map = rps[RP_GROUP_0] << ADF_CFG_SERV_RING_PAIR_0_SHIFT |
rps[RP_GROUP_1] << ADF_CFG_SERV_RING_PAIR_1_SHIFT |
rps[RP_GROUP_0] << ADF_CFG_SERV_RING_PAIR_2_SHIFT |
rps[RP_GROUP_1] << ADF_CFG_SERV_RING_PAIR_3_SHIFT;
return ring_to_svc_map;
}
EXPORT_SYMBOL_GPL(adf_gen4_get_ring_to_svc_map);

1
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.h
View File

@ -235,5 +235,6 @@ int adf_gen4_ring_pair_reset(struct adf_accel_dev *accel_dev, u32 bank_number);
void adf_gen4_set_msix_default_rttable(struct adf_accel_dev *accel_dev);
void adf_gen4_set_ssm_wdtimer(struct adf_accel_dev *accel_dev);
int adf_gen4_init_thd2arb_map(struct adf_accel_dev *accel_dev);
u16 adf_gen4_get_ring_to_svc_map(struct adf_accel_dev *accel_dev);
#endif

6
drivers/crypto/intel/qat/qat_common/adf_gen4_ras.c
View File

@ -1007,8 +1007,7 @@ static bool adf_handle_spppar_err(struct adf_accel_dev *accel_dev,
static bool adf_handle_ssmcpppar_err(struct adf_accel_dev *accel_dev,
void __iomem *csr, u32 iastatssm)
{
u32 reg = ADF_CSR_RD(csr, ADF_GEN4_SSMCPPERR);
u32 bits_num = BITS_PER_REG(reg);
u32 reg, bits_num = BITS_PER_REG(reg);
bool reset_required = false;
unsigned long errs_bits;
u32 bit_iterator;
@ -1106,8 +1105,7 @@ static bool adf_handle_rf_parr_err(struct adf_accel_dev *accel_dev,
static bool adf_handle_ser_err_ssmsh(struct adf_accel_dev *accel_dev,
void __iomem *csr, u32 iastatssm)
{
u32 reg = ADF_CSR_RD(csr, ADF_GEN4_SER_ERR_SSMSH);
u32 bits_num = BITS_PER_REG(reg);
u32 reg, bits_num = BITS_PER_REG(reg);
bool reset_required = false;
unsigned long errs_bits;
u32 bit_iterator;

20
drivers/crypto/intel/qat/qat_common/adf_heartbeat.c
View File

@ -23,12 +23,6 @@
#define ADF_HB_EMPTY_SIG 0xA5A5A5A5
/* Heartbeat counter pair */
struct hb_cnt_pair {
__u16 resp_heartbeat_cnt;
__u16 req_heartbeat_cnt;
};
static int adf_hb_check_polling_freq(struct adf_accel_dev *accel_dev)
{
u64 curr_time = adf_clock_get_current_time();
@ -211,6 +205,19 @@ static int adf_hb_get_status(struct adf_accel_dev *accel_dev)
return ret;
}
static void adf_heartbeat_reset(struct adf_accel_dev *accel_dev)
{
u64 curr_time = adf_clock_get_current_time();
u64 time_since_reset = curr_time - accel_dev->heartbeat->last_hb_reset_time;
if (time_since_reset < ADF_CFG_HB_RESET_MS)
return;
accel_dev->heartbeat->last_hb_reset_time = curr_time;
if (adf_notify_fatal_error(accel_dev))
dev_err(&GET_DEV(accel_dev), "Failed to notify fatal error\n");
}
void adf_heartbeat_status(struct adf_accel_dev *accel_dev,
enum adf_device_heartbeat_status *hb_status)
{
@ -235,6 +242,7 @@ void adf_heartbeat_status(struct adf_accel_dev *accel_dev,
"Heartbeat ERROR: QAT is not responding.\n");
*hb_status = HB_DEV_UNRESPONSIVE;
hb->hb_failed_counter++;
adf_heartbeat_reset(accel_dev);
return;
}

21
drivers/crypto/intel/qat/qat_common/adf_heartbeat.h
View File

@ -13,17 +13,26 @@ struct dentry;
#define ADF_CFG_HB_TIMER_DEFAULT_MS 500
#define ADF_CFG_HB_COUNT_THRESHOLD 3
#define ADF_CFG_HB_RESET_MS 5000
enum adf_device_heartbeat_status {
HB_DEV_UNRESPONSIVE = 0,
HB_DEV_ALIVE,
HB_DEV_UNSUPPORTED,
};
/* Heartbeat counter pair */
struct hb_cnt_pair {
__u16 resp_heartbeat_cnt;
__u16 req_heartbeat_cnt;
};
struct adf_heartbeat {
unsigned int hb_sent_counter;
unsigned int hb_failed_counter;
unsigned int hb_timer;
u64 last_hb_check_time;
u64 last_hb_reset_time;
bool ctrs_cnt_checked;
struct hb_dma_addr {
dma_addr_t phy_addr;
@ -35,6 +44,9 @@ struct adf_heartbeat {
struct dentry *cfg;
struct dentry *sent;
struct dentry *failed;
#ifdef CONFIG_CRYPTO_DEV_QAT_ERROR_INJECTION
struct dentry *inject_error;
#endif
} dbgfs;
};
@ -51,6 +63,15 @@ void adf_heartbeat_status(struct adf_accel_dev *accel_dev,
enum adf_device_heartbeat_status *hb_status);
void adf_heartbeat_check_ctrs(struct adf_accel_dev *accel_dev);
#ifdef CONFIG_CRYPTO_DEV_QAT_ERROR_INJECTION
int adf_heartbeat_inject_error(struct adf_accel_dev *accel_dev);
#else
static inline int adf_heartbeat_inject_error(struct adf_accel_dev *accel_dev)
{
return -EPERM;
}
#endif
#else
static inline int adf_heartbeat_init(struct adf_accel_dev *accel_dev)
{

53
drivers/crypto/intel/qat/qat_common/adf_heartbeat_dbgfs.c
View File

@ -155,6 +155,44 @@ static const struct file_operations adf_hb_cfg_fops = {
.write = adf_hb_cfg_write,
};
static ssize_t adf_hb_error_inject_write(struct file *file,
const char __user *user_buf,
size_t count, loff_t *ppos)
{
struct adf_accel_dev *accel_dev = file->private_data;
char buf[3];
int ret;
/* last byte left as string termination */
if (*ppos != 0 || count != 2)
return -EINVAL;
if (copy_from_user(buf, user_buf, count))
return -EFAULT;
buf[count] = '\0';
if (buf[0] != '1')
return -EINVAL;
ret = adf_heartbeat_inject_error(accel_dev);
if (ret) {
dev_err(&GET_DEV(accel_dev),
"Heartbeat error injection failed with status %d\n",
ret);
return ret;
}
dev_info(&GET_DEV(accel_dev), "Heartbeat error injection enabled\n");
return count;
}
static const struct file_operations adf_hb_error_inject_fops = {
.owner = THIS_MODULE,
.open = simple_open,
.write = adf_hb_error_inject_write,
};
void adf_heartbeat_dbgfs_add(struct adf_accel_dev *accel_dev)
{
struct adf_heartbeat *hb = accel_dev->heartbeat;
@ -171,6 +209,17 @@ void adf_heartbeat_dbgfs_add(struct adf_accel_dev *accel_dev)
&hb->hb_failed_counter, &adf_hb_stats_fops);
hb->dbgfs.cfg = debugfs_create_file("config", 0600, hb->dbgfs.base_dir,
accel_dev, &adf_hb_cfg_fops);
if (IS_ENABLED(CONFIG_CRYPTO_DEV_QAT_ERROR_INJECTION)) {
struct dentry *inject_error __maybe_unused;
inject_error = debugfs_create_file("inject_error", 0200,
hb->dbgfs.base_dir, accel_dev,
&adf_hb_error_inject_fops);
#ifdef CONFIG_CRYPTO_DEV_QAT_ERROR_INJECTION
hb->dbgfs.inject_error = inject_error;
#endif
}
}
EXPORT_SYMBOL_GPL(adf_heartbeat_dbgfs_add);
@ -189,6 +238,10 @@ void adf_heartbeat_dbgfs_rm(struct adf_accel_dev *accel_dev)
hb->dbgfs.failed = NULL;
debugfs_remove(hb->dbgfs.cfg);
hb->dbgfs.cfg = NULL;
#ifdef CONFIG_CRYPTO_DEV_QAT_ERROR_INJECTION
debugfs_remove(hb->dbgfs.inject_error);
hb->dbgfs.inject_error = NULL;
#endif
debugfs_remove(hb->dbgfs.base_dir);
hb->dbgfs.base_dir = NULL;
}

76
drivers/crypto/intel/qat/qat_common/adf_heartbeat_inject.c Normal file
View File

@ -0,0 +1,76 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2023 Intel Corporation */
#include <linux/random.h>
#include "adf_admin.h"
#include "adf_common_drv.h"
#include "adf_heartbeat.h"
#define MAX_HB_TICKS 0xFFFFFFFF
static int adf_hb_set_timer_to_max(struct adf_accel_dev *accel_dev)
{
struct adf_hw_device_data *hw_data = accel_dev->hw_device;
accel_dev->heartbeat->hb_timer = 0;
if (hw_data->stop_timer)
hw_data->stop_timer(accel_dev);
return adf_send_admin_hb_timer(accel_dev, MAX_HB_TICKS);
}
static void adf_set_hb_counters_fail(struct adf_accel_dev *accel_dev, u32 ae,
u32 thr)
{
struct hb_cnt_pair *stats = accel_dev->heartbeat->dma.virt_addr;
struct adf_hw_device_data *hw_device = accel_dev->hw_device;
const size_t max_aes = hw_device->get_num_aes(hw_device);
const size_t hb_ctrs = hw_device->num_hb_ctrs;
size_t thr_id = ae * hb_ctrs + thr;
u16 num_rsp = stats[thr_id].resp_heartbeat_cnt;
/*
* Inject live.req != live.rsp and live.rsp == last.rsp
* to trigger the heartbeat error detection
*/
stats[thr_id].req_heartbeat_cnt++;
stats += (max_aes * hb_ctrs);
stats[thr_id].resp_heartbeat_cnt = num_rsp;
}
int adf_heartbeat_inject_error(struct adf_accel_dev *accel_dev)
{
struct adf_hw_device_data *hw_device = accel_dev->hw_device;
const size_t max_aes = hw_device->get_num_aes(hw_device);
const size_t hb_ctrs = hw_device->num_hb_ctrs;
u32 rand, rand_ae, rand_thr;
unsigned long ae_mask;
int ret;
ae_mask = hw_device->ae_mask;
do {
/* Ensure we have a valid ae */
get_random_bytes(&rand, sizeof(rand));
rand_ae = rand % max_aes;
} while (!test_bit(rand_ae, &ae_mask));
get_random_bytes(&rand, sizeof(rand));
rand_thr = rand % hb_ctrs;
/* Increase the heartbeat timer to prevent FW updating HB counters */
ret = adf_hb_set_timer_to_max(accel_dev);
if (ret)
return ret;
/* Configure worker threads to stop processing any packet */
ret = adf_disable_arb_thd(accel_dev, rand_ae, rand_thr);
if (ret)
return ret;
/* Change HB counters memory to simulate a hang */
adf_set_hb_counters_fail(accel_dev, rand_ae, rand_thr);
return 0;
}

25
drivers/crypto/intel/qat/qat_common/adf_hw_arbiter.c
View File

@ -103,3 +103,28 @@ void adf_exit_arb(struct adf_accel_dev *accel_dev)
csr_ops->write_csr_ring_srv_arb_en(csr, i, 0);
}
EXPORT_SYMBOL_GPL(adf_exit_arb);
int adf_disable_arb_thd(struct adf_accel_dev *accel_dev, u32 ae, u32 thr)
{
void __iomem *csr = accel_dev->transport->banks[0].csr_addr;
struct adf_hw_device_data *hw_data = accel_dev->hw_device;
const u32 *thd_2_arb_cfg;
struct arb_info info;
u32 ae_thr_map;
if (ADF_AE_STRAND0_THREAD == thr || ADF_AE_STRAND1_THREAD == thr)
thr = ADF_AE_ADMIN_THREAD;
hw_data->get_arb_info(&info);
thd_2_arb_cfg = hw_data->get_arb_mapping(accel_dev);
if (!thd_2_arb_cfg)
return -EFAULT;
/* Disable scheduling for this particular AE and thread */
ae_thr_map = *(thd_2_arb_cfg + ae);
ae_thr_map &= ~(GENMASK(3, 0) << (thr * BIT(2)));
WRITE_CSR_ARB_WT2SAM(csr, info.arb_offset, info.wt2sam_offset, ae,
ae_thr_map);
return 0;
}

12
drivers/crypto/intel/qat/qat_common/adf_init.c
View File

@ -433,6 +433,18 @@ int adf_dev_restarted_notify(struct adf_accel_dev *accel_dev)
return 0;
}
void adf_error_notifier(struct adf_accel_dev *accel_dev)
{
struct service_hndl *service;
list_for_each_entry(service, &service_table, list) {
if (service->event_hld(accel_dev, ADF_EVENT_FATAL_ERROR))
dev_err(&GET_DEV(accel_dev),
"Failed to send error event to %s.\n",
service->name);
}
}
static int adf_dev_shutdown_cache_cfg(struct adf_accel_dev *accel_dev)
{
char services[ADF_CFG_MAX_VAL_LEN_IN_BYTES] = {0};

11
drivers/crypto/intel/qat/qat_common/adf_isr.c
View File

@ -139,8 +139,13 @@ static bool adf_handle_ras_int(struct adf_accel_dev *accel_dev)
if (ras_ops->handle_interrupt &&
ras_ops->handle_interrupt(accel_dev, &reset_required)) {
if (reset_required)
if (reset_required) {
dev_err(&GET_DEV(accel_dev), "Fatal error, reset required\n");
if (adf_notify_fatal_error(accel_dev))
dev_err(&GET_DEV(accel_dev),
"Failed to notify fatal error\n");
}
return true;
}
@ -272,7 +277,7 @@ static int adf_isr_alloc_msix_vectors_data(struct adf_accel_dev *accel_dev)
if (!accel_dev->pf.vf_info)
msix_num_entries += hw_data->num_banks;
irqs = kzalloc_node(msix_num_entries * sizeof(*irqs),
irqs = kcalloc_node(msix_num_entries, sizeof(*irqs),
GFP_KERNEL, dev_to_node(&GET_DEV(accel_dev)));
if (!irqs)
return -ENOMEM;
@ -375,8 +380,6 @@ EXPORT_SYMBOL_GPL(adf_isr_resource_alloc);
/**
* adf_init_misc_wq() - Init misc workqueue
*
* Function init workqueue 'qat_misc_wq' for general purpose.
*
* Return: 0 on success, error code otherwise.
*/
int __init adf_init_misc_wq(void)

7
drivers/crypto/intel/qat/qat_common/adf_pfvf_msg.h
View File

@ -99,6 +99,8 @@ enum pf2vf_msgtype {
ADF_PF2VF_MSGTYPE_RESTARTING = 0x01,
ADF_PF2VF_MSGTYPE_VERSION_RESP = 0x02,
ADF_PF2VF_MSGTYPE_BLKMSG_RESP = 0x03,
ADF_PF2VF_MSGTYPE_FATAL_ERROR = 0x04,
ADF_PF2VF_MSGTYPE_RESTARTED = 0x05,
/* Values from 0x10 are Gen4 specific, message type is only 4 bits in Gen2 devices. */
ADF_PF2VF_MSGTYPE_RP_RESET_RESP = 0x10,
};
@ -112,6 +114,7 @@ enum vf2pf_msgtype {
ADF_VF2PF_MSGTYPE_LARGE_BLOCK_REQ = 0x07,
ADF_VF2PF_MSGTYPE_MEDIUM_BLOCK_REQ = 0x08,
ADF_VF2PF_MSGTYPE_SMALL_BLOCK_REQ = 0x09,
ADF_VF2PF_MSGTYPE_RESTARTING_COMPLETE = 0x0a,
/* Values from 0x10 are Gen4 specific, message type is only 4 bits in Gen2 devices. */
ADF_VF2PF_MSGTYPE_RP_RESET = 0x10,
};
@ -124,8 +127,10 @@ enum pfvf_compatibility_version {
ADF_PFVF_COMPAT_FAST_ACK = 0x03,
/* Ring to service mapping support for non-standard mappings */
ADF_PFVF_COMPAT_RING_TO_SVC_MAP = 0x04,
/* Fallback compat */
ADF_PFVF_COMPAT_FALLBACK = 0x05,
/* Reference to the latest version */
ADF_PFVF_COMPAT_THIS_VERSION = 0x04,
ADF_PFVF_COMPAT_THIS_VERSION = 0x05,
};
/* PF->VF Version Response */

64
drivers/crypto/intel/qat/qat_common/adf_pfvf_pf_msg.c
View File

@ -1,21 +1,83 @@
// SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0-only)
/* Copyright(c) 2015 - 2021 Intel Corporation */
#include <linux/delay.h>
#include <linux/pci.h>
#include "adf_accel_devices.h"
#include "adf_pfvf_msg.h"
#include "adf_pfvf_pf_msg.h"
#include "adf_pfvf_pf_proto.h"
#define ADF_PF_WAIT_RESTARTING_COMPLETE_DELAY 100
#define ADF_VF_SHUTDOWN_RETRY 100
void adf_pf2vf_notify_restarting(struct adf_accel_dev *accel_dev)
{
struct adf_accel_vf_info *vf;
struct pfvf_message msg = { .type = ADF_PF2VF_MSGTYPE_RESTARTING };
int i, num_vfs = pci_num_vf(accel_to_pci_dev(accel_dev));
dev_dbg(&GET_DEV(accel_dev), "pf2vf notify restarting\n");
for (i = 0, vf = accel_dev->pf.vf_info; i < num_vfs; i++, vf++) {
if (vf->init && adf_send_pf2vf_msg(accel_dev, i, msg))
vf->restarting = false;
if (!vf->init)
continue;
if (adf_send_pf2vf_msg(accel_dev, i, msg))
dev_err(&GET_DEV(accel_dev),
"Failed to send restarting msg to VF%d\n", i);
else if (vf->vf_compat_ver >= ADF_PFVF_COMPAT_FALLBACK)
vf->restarting = true;
}
}
void adf_pf2vf_wait_for_restarting_complete(struct adf_accel_dev *accel_dev)
{
int num_vfs = pci_num_vf(accel_to_pci_dev(accel_dev));
int i, retries = ADF_VF_SHUTDOWN_RETRY;
struct adf_accel_vf_info *vf;
bool vf_running;
dev_dbg(&GET_DEV(accel_dev), "pf2vf wait for restarting complete\n");
do {
vf_running = false;
for (i = 0, vf = accel_dev->pf.vf_info; i < num_vfs; i++, vf++)
if (vf->restarting)
vf_running = true;
if (!vf_running)
break;
msleep(ADF_PF_WAIT_RESTARTING_COMPLETE_DELAY);
} while (--retries);
if (vf_running)
dev_warn(&GET_DEV(accel_dev), "Some VFs are still running\n");
}
void adf_pf2vf_notify_restarted(struct adf_accel_dev *accel_dev)
{
struct pfvf_message msg = { .type = ADF_PF2VF_MSGTYPE_RESTARTED };
int i, num_vfs = pci_num_vf(accel_to_pci_dev(accel_dev));
struct adf_accel_vf_info *vf;
dev_dbg(&GET_DEV(accel_dev), "pf2vf notify restarted\n");
for (i = 0, vf = accel_dev->pf.vf_info; i < num_vfs; i++, vf++) {
if (vf->init && vf->vf_compat_ver >= ADF_PFVF_COMPAT_FALLBACK &&
adf_send_pf2vf_msg(accel_dev, i, msg))
dev_err(&GET_DEV(accel_dev),
"Failed to send restarted msg to VF%d\n", i);
}
}
void adf_pf2vf_notify_fatal_error(struct adf_accel_dev *accel_dev)
{
struct pfvf_message msg = { .type = ADF_PF2VF_MSGTYPE_FATAL_ERROR };
int i, num_vfs = pci_num_vf(accel_to_pci_dev(accel_dev));
struct adf_accel_vf_info *vf;
dev_dbg(&GET_DEV(accel_dev), "pf2vf notify fatal error\n");
for (i = 0, vf = accel_dev->pf.vf_info; i < num_vfs; i++, vf++) {
if (vf->init && vf->vf_compat_ver >= ADF_PFVF_COMPAT_FALLBACK &&
adf_send_pf2vf_msg(accel_dev, i, msg))
dev_err(&GET_DEV(accel_dev),
"Failed to send fatal error msg to VF%d\n", i);
}
}

21
drivers/crypto/intel/qat/qat_common/adf_pfvf_pf_msg.h
View File

@ -5,7 +5,28 @@
#include "adf_accel_devices.h"
#if defined(CONFIG_PCI_IOV)
void adf_pf2vf_notify_restarting(struct adf_accel_dev *accel_dev);
void adf_pf2vf_wait_for_restarting_complete(struct adf_accel_dev *accel_dev);
void adf_pf2vf_notify_restarted(struct adf_accel_dev *accel_dev);
void adf_pf2vf_notify_fatal_error(struct adf_accel_dev *accel_dev);
#else
static inline void adf_pf2vf_notify_restarting(struct adf_accel_dev *accel_dev)
{
}
static inline void adf_pf2vf_wait_for_restarting_complete(struct adf_accel_dev *accel_dev)
{
}
static inline void adf_pf2vf_notify_restarted(struct adf_accel_dev *accel_dev)
{
}
static inline void adf_pf2vf_notify_fatal_error(struct adf_accel_dev *accel_dev)
{
}
#endif
typedef int (*adf_pf2vf_blkmsg_provider)(struct adf_accel_dev *accel_dev,
u8 *buffer, u8 compat);

8
drivers/crypto/intel/qat/qat_common/adf_pfvf_pf_proto.c
View File

@ -291,6 +291,14 @@ static int adf_handle_vf2pf_msg(struct adf_accel_dev *accel_dev, u8 vf_nr,
vf_info->init = false;
}
break;
case ADF_VF2PF_MSGTYPE_RESTARTING_COMPLETE:
{
dev_dbg(&GET_DEV(accel_dev),
"Restarting Complete received from VF%d\n", vf_nr);
vf_info->restarting = false;
vf_info->init = false;
}
break;
case ADF_VF2PF_MSGTYPE_LARGE_BLOCK_REQ:
case ADF_VF2PF_MSGTYPE_MEDIUM_BLOCK_REQ:
case ADF_VF2PF_MSGTYPE_SMALL_BLOCK_REQ:

6
drivers/crypto/intel/qat/qat_common/adf_pfvf_vf_proto.c
View File

@ -308,6 +308,12 @@ static bool adf_handle_pf2vf_msg(struct adf_accel_dev *accel_dev,
adf_pf2vf_handle_pf_restarting(accel_dev);
return false;
case ADF_PF2VF_MSGTYPE_RESTARTED:
dev_dbg(&GET_DEV(accel_dev), "Restarted message received from PF\n");
return true;
case ADF_PF2VF_MSGTYPE_FATAL_ERROR:
dev_err(&GET_DEV(accel_dev), "Fatal error received from PF\n");
return true;
case ADF_PF2VF_MSGTYPE_VERSION_RESP:
case ADF_PF2VF_MSGTYPE_BLKMSG_RESP:
case ADF_PF2VF_MSGTYPE_RP_RESET_RESP:

20
drivers/crypto/intel/qat/qat_common/adf_rl.c
View File

@ -788,6 +788,24 @@ static void clear_sla(struct adf_rl *rl_data, struct rl_sla *sla)
sla_type_arr[node_id] = NULL;
}
static void free_all_sla(struct adf_accel_dev *accel_dev)
{
struct adf_rl *rl_data = accel_dev->rate_limiting;
int sla_id;
mutex_lock(&rl_data->rl_lock);
for (sla_id = 0; sla_id < RL_NODES_CNT_MAX; sla_id++) {
if (!rl_data->sla[sla_id])
continue;
kfree(rl_data->sla[sla_id]);
rl_data->sla[sla_id] = NULL;
}
mutex_unlock(&rl_data->rl_lock);
}
/**
* add_update_sla() - handles the creation and the update of an SLA
* @accel_dev: pointer to acceleration device structure
@ -1155,7 +1173,7 @@ void adf_rl_stop(struct adf_accel_dev *accel_dev)
return;
adf_sysfs_rl_rm(accel_dev);
adf_rl_remove_sla_all(accel_dev, true);
free_all_sla(accel_dev);
}
void adf_rl_exit(struct adf_accel_dev *accel_dev)

38
drivers/crypto/intel/qat/qat_common/adf_sriov.c
View File

@ -60,7 +60,6 @@ static int adf_enable_sriov(struct adf_accel_dev *accel_dev)
/* This ptr will be populated when VFs will be created */
vf_info->accel_dev = accel_dev;
vf_info->vf_nr = i;
vf_info->vf_compat_ver = 0;
mutex_init(&vf_info->pf2vf_lock);
ratelimit_state_init(&vf_info->vf2pf_ratelimit,
@ -84,6 +83,32 @@ static int adf_enable_sriov(struct adf_accel_dev *accel_dev)
return pci_enable_sriov(pdev, totalvfs);
}
void adf_reenable_sriov(struct adf_accel_dev *accel_dev)
{
struct pci_dev *pdev = accel_to_pci_dev(accel_dev);
char cfg[ADF_CFG_MAX_VAL_LEN_IN_BYTES] = {0};
unsigned long val = 0;
if (adf_cfg_get_param_value(accel_dev, ADF_GENERAL_SEC,
ADF_SRIOV_ENABLED, cfg))
return;
if (!accel_dev->pf.vf_info)
return;
if (adf_cfg_add_key_value_param(accel_dev, ADF_KERNEL_SEC, ADF_NUM_CY,
&val, ADF_DEC))
return;
if (adf_cfg_add_key_value_param(accel_dev, ADF_KERNEL_SEC, ADF_NUM_DC,
&val, ADF_DEC))
return;
set_bit(ADF_STATUS_CONFIGURED, &accel_dev->status);
dev_dbg(&pdev->dev, "Re-enabling SRIOV\n");
adf_enable_sriov(accel_dev);
}
/**
* adf_disable_sriov() - Disable SRIOV for the device
* @accel_dev: Pointer to accel device.
@ -103,6 +128,7 @@ void adf_disable_sriov(struct adf_accel_dev *accel_dev)
return;
adf_pf2vf_notify_restarting(accel_dev);
adf_pf2vf_wait_for_restarting_complete(accel_dev);
pci_disable_sriov(accel_to_pci_dev(accel_dev));
/* Disable VF to PF interrupts */
@ -115,8 +141,10 @@ void adf_disable_sriov(struct adf_accel_dev *accel_dev)
for (i = 0, vf = accel_dev->pf.vf_info; i < totalvfs; i++, vf++)
mutex_destroy(&vf->pf2vf_lock);
kfree(accel_dev->pf.vf_info);
accel_dev->pf.vf_info = NULL;
if (!test_bit(ADF_STATUS_RESTARTING, &accel_dev->status)) {
kfree(accel_dev->pf.vf_info);
accel_dev->pf.vf_info = NULL;
}
}
EXPORT_SYMBOL_GPL(adf_disable_sriov);
@ -194,6 +222,10 @@ int adf_sriov_configure(struct pci_dev *pdev, int numvfs)
if (ret)
return ret;
val = 1;
adf_cfg_add_key_value_param(accel_dev, ADF_GENERAL_SEC, ADF_SRIOV_ENABLED,
&val, ADF_DEC);
return numvfs;
}
EXPORT_SYMBOL_GPL(adf_sriov_configure);

37
drivers/crypto/intel/qat/qat_common/adf_sysfs.c
View File

@ -204,6 +204,42 @@ static ssize_t pm_idle_enabled_store(struct device *dev, struct device_attribute
}
static DEVICE_ATTR_RW(pm_idle_enabled);
static ssize_t auto_reset_show(struct device *dev, struct device_attribute *attr,
char *buf)
{
char *auto_reset;
struct adf_accel_dev *accel_dev;
accel_dev = adf_devmgr_pci_to_accel_dev(to_pci_dev(dev));
if (!accel_dev)
return -EINVAL;
auto_reset = accel_dev->autoreset_on_error ? "on" : "off";
return sysfs_emit(buf, "%s\n", auto_reset);
}
static ssize_t auto_reset_store(struct device *dev, struct device_attribute *attr,
const char *buf, size_t count)
{
struct adf_accel_dev *accel_dev;
bool enabled = false;
int ret;
ret = kstrtobool(buf, &enabled);
if (ret)
return ret;
accel_dev = adf_devmgr_pci_to_accel_dev(to_pci_dev(dev));
if (!accel_dev)
return -EINVAL;
accel_dev->autoreset_on_error = enabled;
return count;
}
static DEVICE_ATTR_RW(auto_reset);
static DEVICE_ATTR_RW(state);
static DEVICE_ATTR_RW(cfg_services);
@ -291,6 +327,7 @@ static struct attribute *qat_attrs[] = {
&dev_attr_pm_idle_enabled.attr,
&dev_attr_rp2srv.attr,
&dev_attr_num_rps.attr,
&dev_attr_auto_reset.attr,
NULL,
};

2
drivers/crypto/intel/qat/qat_common/adf_vf_isr.c
View File

@ -293,8 +293,6 @@ EXPORT_SYMBOL_GPL(adf_flush_vf_wq);
/**
* adf_init_vf_wq() - Init workqueue for VF
*
* Function init workqueue 'adf_vf_stop_wq' for VF.
*
* Return: 0 on success, error code otherwise.
*/
int __init adf_init_vf_wq(void)

9
drivers/crypto/intel/qat/qat_common/qat_comp_algs.c
View File

@ -13,15 +13,6 @@
#include "qat_compression.h"
#include "qat_algs_send.h"
#define QAT_RFC_1950_HDR_SIZE 2
#define QAT_RFC_1950_FOOTER_SIZE 4
#define QAT_RFC_1950_CM_DEFLATE 8
#define QAT_RFC_1950_CM_DEFLATE_CINFO_32K 7
#define QAT_RFC_1950_CM_MASK 0x0f
#define QAT_RFC_1950_CM_OFFSET 4
#define QAT_RFC_1950_DICT_MASK 0x20
#define QAT_RFC_1950_COMP_HDR 0x785e
static DEFINE_MUTEX(algs_lock);
static unsigned int active_devs;

4
drivers/crypto/intel/qat/qat_common/qat_crypto.c
View File

@ -105,8 +105,8 @@ struct qat_crypto_instance *qat_crypto_get_instance_node(int node)
}
/**
* qat_crypto_vf_dev_config()
* create dev config required to create crypto inst.
* qat_crypto_vf_dev_config() - create dev config required to create
* crypto inst.
*
* @accel_dev: Pointer to acceleration device.
*

5
drivers/crypto/rockchip/rk3288_crypto.c
View File

@ -371,6 +371,11 @@ static int rk_crypto_probe(struct platform_device *pdev)
}
crypto_info->engine = crypto_engine_alloc_init(&pdev->dev, true);
if (!crypto_info->engine) {
err = -ENOMEM;
goto err_crypto;
}
crypto_engine_start(crypto_info->engine);
init_completion(&crypto_info->complete);

12
drivers/crypto/virtio/virtio_crypto_akcipher_algs.c
View File

@ -225,11 +225,11 @@ static int __virtio_crypto_akcipher_do_req(struct virtio_crypto_akcipher_request
struct virtio_crypto *vcrypto = ctx->vcrypto;
struct virtio_crypto_op_data_req *req_data = vc_req->req_data;
struct scatterlist *sgs[4], outhdr_sg, inhdr_sg, srcdata_sg, dstdata_sg;
void *src_buf = NULL, *dst_buf = NULL;
void *src_buf, *dst_buf = NULL;
unsigned int num_out = 0, num_in = 0;
int node = dev_to_node(&vcrypto->vdev->dev);
unsigned long flags;
int ret = -ENOMEM;
int ret;
bool verify = vc_akcipher_req->opcode == VIRTIO_CRYPTO_AKCIPHER_VERIFY;
unsigned int src_len = verify ? req->src_len + req->dst_len : req->src_len;
@ -240,7 +240,7 @@ static int __virtio_crypto_akcipher_do_req(struct virtio_crypto_akcipher_request
/* src data */
src_buf = kcalloc_node(src_len, 1, GFP_KERNEL, node);
if (!src_buf)
goto err;
return -ENOMEM;
if (verify) {
/* for verify operation, both src and dst data work as OUT direction */
@ -255,7 +255,7 @@ static int __virtio_crypto_akcipher_do_req(struct virtio_crypto_akcipher_request
/* dst data */
dst_buf = kcalloc_node(req->dst_len, 1, GFP_KERNEL, node);
if (!dst_buf)
goto err;
goto free_src;
sg_init_one(&dstdata_sg, dst_buf, req->dst_len);
sgs[num_out + num_in++] = &dstdata_sg;
@ -278,9 +278,9 @@ static int __virtio_crypto_akcipher_do_req(struct virtio_crypto_akcipher_request
return 0;
err:
kfree(src_buf);
kfree(dst_buf);
free_src:
kfree(src_buf);
return -ENOMEM;
}

2
drivers/crypto/virtio/virtio_crypto_core.c
View File

@ -42,8 +42,6 @@ static void virtcrypto_ctrlq_callback(struct virtqueue *vq)
virtio_crypto_ctrlq_callback(vc_ctrl_req);
spin_lock_irqsave(&vcrypto->ctrl_lock, flags);
}
if (unlikely(virtqueue_is_broken(vq)))
break;
} while (!virtqueue_enable_cb(vq));
spin_unlock_irqrestore(&vcrypto->ctrl_lock, flags);
}

3
drivers/crypto/vmx/.gitignore vendored
View File

@ -1,3 +0,0 @@
# SPDX-License-Identifier: GPL-2.0-only
aesp8-ppc.S
ghashp8-ppc.S

14
drivers/crypto/vmx/Kconfig
View File

@ -1,14 +0,0 @@
# SPDX-License-Identifier: GPL-2.0-only
config CRYPTO_DEV_VMX_ENCRYPT
tristate "Encryption acceleration support on P8 CPU"
depends on CRYPTO_DEV_VMX
select CRYPTO_AES
select CRYPTO_CBC
select CRYPTO_CTR
select CRYPTO_GHASH
select CRYPTO_XTS
default m
help
Support for VMX cryptographic acceleration instructions on Power8 CPU.
This module supports acceleration for AES and GHASH in hardware. If you
choose 'M' here, this module will be called vmx-crypto.

23
drivers/crypto/vmx/Makefile
View File

@ -1,23 +0,0 @@
# SPDX-License-Identifier: GPL-2.0
obj-$(CONFIG_CRYPTO_DEV_VMX_ENCRYPT) += vmx-crypto.o
vmx-crypto-objs := vmx.o aesp8-ppc.o ghashp8-ppc.o aes.o aes_cbc.o aes_ctr.o aes_xts.o ghash.o
ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y)
override flavour := linux-ppc64le
else
ifdef CONFIG_PPC64_ELF_ABI_V2
override flavour := linux-ppc64-elfv2
else
override flavour := linux-ppc64
endif
endif
quiet_cmd_perl = PERL $@
cmd_perl = $(PERL) $< $(flavour) > $@
targets += aesp8-ppc.S ghashp8-ppc.S
$(obj)/aesp8-ppc.S $(obj)/ghashp8-ppc.S: $(obj)/%.S: $(src)/%.pl FORCE
$(call if_changed,perl)
OBJECT_FILES_NON_STANDARD_aesp8-ppc.o := y

231
drivers/crypto/vmx/ppc-xlate.pl
View File

@ -1,231 +0,0 @@
#!/usr/bin/env perl
# SPDX-License-Identifier: GPL-2.0
# PowerPC assembler distiller by <appro>.
my $flavour = shift;
my $output = shift;
open STDOUT,">$output" || die "can't open $output: $!";
my %GLOBALS;
my $dotinlocallabels=($flavour=~/linux/)?1:0;
my $elfv2abi=(($flavour =~ /linux-ppc64le/) or ($flavour =~ /linux-ppc64-elfv2/))?1:0;
my $dotfunctions=($elfv2abi=~1)?0:1;
################################################################
# directives which need special treatment on different platforms
################################################################
my $globl = sub {
my $junk = shift;
my $name = shift;
my $global = \$GLOBALS{$name};
my $ret;
$name =~ s|^[\.\_]||;
SWITCH: for ($flavour) {
/aix/ && do { $name = ".$name";
last;
};
/osx/ && do { $name = "_$name";
last;
};
/linux/
&& do { $ret = "_GLOBAL($name)";
last;
};
}
$ret = ".globl $name\nalign 5\n$name:" if (!$ret);
$$global = $name;
$ret;
};
my $text = sub {
my $ret = ($flavour =~ /aix/) ? ".csect\t.text[PR],7" : ".text";
$ret = ".abiversion 2\n".$ret if ($elfv2abi);
$ret;
};
my $machine = sub {
my $junk = shift;
my $arch = shift;
if ($flavour =~ /osx/)
{ $arch =~ s/\"//g;
$arch = ($flavour=~/64/) ? "ppc970-64" : "ppc970" if ($arch eq "any");
}
".machine $arch";
};
my $size = sub {
if ($flavour =~ /linux/)
{ shift;
my $name = shift; $name =~ s|^[\.\_]||;
my $ret = ".size $name,.-".($dotfunctions?".":"").$name;
$ret .= "\n.size .$name,.-.$name" if ($dotfunctions);
$ret;
}
else
{ ""; }
};
my $asciz = sub {
shift;
my $line = join(",",@_);
if ($line =~ /^"(.*)"$/)
{ ".byte " . join(",",unpack("C*",$1),0) . "\n.align 2"; }
else
{ ""; }
};
my $quad = sub {
shift;
my @ret;
my ($hi,$lo);
for (@_) {
if (/^0x([0-9a-f]*?)([0-9a-f]{1,8})$/io)
{ $hi=$1?"0x$1":"0"; $lo="0x$2"; }
elsif (/^([0-9]+)$/o)
{ $hi=$1>>32; $lo=$1&0xffffffff; } # error-prone with 32-bit perl
else
{ $hi=undef; $lo=$_; }
if (defined($hi))
{ push(@ret,$flavour=~/le$/o?".long\t$lo,$hi":".long\t$hi,$lo"); }
else
{ push(@ret,".quad $lo"); }
}
join("\n",@ret);
};
################################################################
# simplified mnemonics not handled by at least one assembler
################################################################
my $cmplw = sub {
my $f = shift;
my $cr = 0; $cr = shift if ($#_>1);
# Some out-of-date 32-bit GNU assembler just can't handle cmplw...
($flavour =~ /linux.*32/) ?
" .long ".sprintf "0x%x",31<<26|$cr<<23|$_[0]<<16|$_[1]<<11|64 :
" cmplw ".join(',',$cr,@_);
};
my $bdnz = sub {
my $f = shift;
my $bo = $f=~/[\+\-]/ ? 16+9 : 16; # optional "to be taken" hint
" bc $bo,0,".shift;
} if ($flavour!~/linux/);
my $bltlr = sub {
my $f = shift;
my $bo = $f=~/\-/ ? 12+2 : 12; # optional "not to be taken" hint
($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints
" .long ".sprintf "0x%x",19<<26|$bo<<21|16<<1 :
" bclr $bo,0";
};
my $bnelr = sub {
my $f = shift;
my $bo = $f=~/\-/ ? 4+2 : 4; # optional "not to be taken" hint
($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints
" .long ".sprintf "0x%x",19<<26|$bo<<21|2<<16|16<<1 :
" bclr $bo,2";
};
my $beqlr = sub {
my $f = shift;
my $bo = $f=~/-/ ? 12+2 : 12; # optional "not to be taken" hint
($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints
" .long ".sprintf "0x%X",19<<26|$bo<<21|2<<16|16<<1 :
" bclr $bo,2";
};
# GNU assembler can't handle extrdi rA,rS,16,48, or when sum of last two
# arguments is 64, with "operand out of range" error.
my $extrdi = sub {
my ($f,$ra,$rs,$n,$b) = @_;
$b = ($b+$n)&63; $n = 64-$n;
" rldicl $ra,$rs,$b,$n";
};
my $vmr = sub {
my ($f,$vx,$vy) = @_;
" vor $vx,$vy,$vy";
};
# Some ABIs specify vrsave, special-purpose register #256, as reserved
# for system use.
my $no_vrsave = ($elfv2abi);
my $mtspr = sub {
my ($f,$idx,$ra) = @_;
if ($idx == 256 && $no_vrsave) {
" or $ra,$ra,$ra";
} else {
" mtspr $idx,$ra";
}
};
my $mfspr = sub {
my ($f,$rd,$idx) = @_;
if ($idx == 256 && $no_vrsave) {
" li $rd,-1";
} else {
" mfspr $rd,$idx";
}
};
# PowerISA 2.06 stuff
sub vsxmem_op {
my ($f, $vrt, $ra, $rb, $op) = @_;
" .long ".sprintf "0x%X",(31<<26)|($vrt<<21)|($ra<<16)|($rb<<11)|($op*2+1);
}
# made-up unaligned memory reference AltiVec/VMX instructions
my $lvx_u = sub { vsxmem_op(@_, 844); }; # lxvd2x
my $stvx_u = sub { vsxmem_op(@_, 972); }; # stxvd2x
my $lvdx_u = sub { vsxmem_op(@_, 588); }; # lxsdx
my $stvdx_u = sub { vsxmem_op(@_, 716); }; # stxsdx
my $lvx_4w = sub { vsxmem_op(@_, 780); }; # lxvw4x
my $stvx_4w = sub { vsxmem_op(@_, 908); }; # stxvw4x
# PowerISA 2.07 stuff
sub vcrypto_op {
my ($f, $vrt, $vra, $vrb, $op) = @_;
" .long ".sprintf "0x%X",(4<<26)|($vrt<<21)|($vra<<16)|($vrb<<11)|$op;
}
my $vcipher = sub { vcrypto_op(@_, 1288); };
my $vcipherlast = sub { vcrypto_op(@_, 1289); };
my $vncipher = sub { vcrypto_op(@_, 1352); };
my $vncipherlast= sub { vcrypto_op(@_, 1353); };
my $vsbox = sub { vcrypto_op(@_, 0, 1480); };
my $vshasigmad = sub { my ($st,$six)=splice(@_,-2); vcrypto_op(@_, $st<<4|$six, 1730); };
my $vshasigmaw = sub { my ($st,$six)=splice(@_,-2); vcrypto_op(@_, $st<<4|$six, 1666); };
my $vpmsumb = sub { vcrypto_op(@_, 1032); };
my $vpmsumd = sub { vcrypto_op(@_, 1224); };
my $vpmsubh = sub { vcrypto_op(@_, 1096); };
my $vpmsumw = sub { vcrypto_op(@_, 1160); };
my $vaddudm = sub { vcrypto_op(@_, 192); };
my $vadduqm = sub { vcrypto_op(@_, 256); };
my $mtsle = sub {
my ($f, $arg) = @_;
" .long ".sprintf "0x%X",(31<<26)|($arg<<21)|(147*2);
};
print "#include <asm/ppc_asm.h>\n" if $flavour =~ /linux/;
while($line=<>) {
$line =~ s|[#!;].*$||; # get rid of asm-style comments...
$line =~ s|/\*.*\*/||; # ... and C-style comments...
$line =~ s|^\s+||; # ... and skip white spaces in beginning...
$line =~ s|\s+$||; # ... and at the end
{
$line =~ s|\b\.L(\w+)|L$1|g; # common denominator for Locallabel
$line =~ s|\bL(\w+)|\.L$1|g if ($dotinlocallabels);
}
{
$line =~ s|^\s*(\.?)(\w+)([\.\+\-]?)\s*||;
my $c = $1; $c = "\t" if ($c eq "");
my $mnemonic = $2;
my $f = $3;
my $opcode = eval("\$$mnemonic");
$line =~ s/\b(c?[rf]|v|vs)([0-9]+)\b/$2/g if ($c ne "." and $flavour !~ /osx/);
if (ref($opcode) eq 'CODE') { $line = &$opcode($f,split(',',$line)); }
elsif ($mnemonic) { $line = $c.$mnemonic.$f."\t".$line; }
}
print $line if ($line);
print "\n";
}
close STDOUT;

3
drivers/crypto/xilinx/zynqmp-aes-gcm.c
View File

@ -231,7 +231,10 @@ static int zynqmp_handle_aes_req(struct crypto_engine *engine,
err = zynqmp_aes_aead_cipher(areq);
}
local_bh_disable();
crypto_finalize_aead_request(engine, areq, err);
local_bh_enable();
return 0;
}

2
include/crypto/internal/hash.h
View File

@ -87,8 +87,6 @@ static inline bool crypto_shash_alg_needs_key(struct shash_alg *alg)
!(alg->base.cra_flags & CRYPTO_ALG_OPTIONAL_KEY);
}
bool crypto_hash_alg_has_setkey(struct hash_alg_common *halg);
int crypto_grab_ahash(struct crypto_ahash_spawn *spawn,
struct crypto_instance *inst,
const char *name, u32 type, u32 mask);

1
include/crypto/public_key.h
View File

@ -10,6 +10,7 @@
#ifndef _LINUX_PUBLIC_KEY_H
#define _LINUX_PUBLIC_KEY_H
#include <linux/errno.h>
#include <linux/keyctl.h>
#include <linux/oid_registry.h>

10
include/linux/hisi_acc_qm.h
View File

@ -43,6 +43,7 @@
#define QM_MB_CMD_CQC_BT 0x5
#define QM_MB_CMD_SQC_VFT_V2 0x6
#define QM_MB_CMD_STOP_QP 0x8
#define QM_MB_CMD_FLUSH_QM 0x9
#define QM_MB_CMD_SRC 0xc
#define QM_MB_CMD_DST 0xd
@ -151,6 +152,7 @@ enum qm_cap_bits {
QM_SUPPORT_DB_ISOLATION = 0x0,
QM_SUPPORT_FUNC_QOS,
QM_SUPPORT_STOP_QP,
QM_SUPPORT_STOP_FUNC,
QM_SUPPORT_MB_COMMAND,
QM_SUPPORT_SVA_PREFETCH,
QM_SUPPORT_RPM,
@ -161,6 +163,11 @@ struct qm_dev_alg {
const char *alg;
};
struct qm_dev_dfx {
u32 dev_state;
u32 dev_timeout;
};
struct dfx_diff_registers {
u32 *regs;
u32 reg_offset;
@ -189,6 +196,7 @@ struct qm_debug {
struct dentry *debug_root;
struct dentry *qm_d;
struct debugfs_file files[DEBUG_FILE_NUM];
struct qm_dev_dfx dev_dfx;
unsigned int *qm_last_words;
/* ACC engines recoreding last regs */
unsigned int *last_words;
@ -523,7 +531,7 @@ void hisi_qm_uninit(struct hisi_qm *qm);
int hisi_qm_start(struct hisi_qm *qm);
int hisi_qm_stop(struct hisi_qm *qm, enum qm_stop_reason r);
int hisi_qm_start_qp(struct hisi_qp *qp, unsigned long arg);
int hisi_qm_stop_qp(struct hisi_qp *qp);
void hisi_qm_stop_qp(struct hisi_qp *qp);
int hisi_qp_send(struct hisi_qp *qp, const void *msg);
void hisi_qm_debug_init(struct hisi_qm *qm);
void hisi_qm_debug_regs_clear(struct hisi_qm *qm);

8
tools/crypto/ccp/test_dbc.py
View File

@ -138,12 +138,14 @@ class TestInvalidSignature(DynamicBoostControlTest):
def test_authenticated_nonce(self) -> None:
"""fetch authenticated nonce"""
get_nonce(self.d, None)
with self.assertRaises(OSError) as error:
get_nonce(self.d, self.signature)
self.assertEqual(error.exception.errno, 1)
self.assertEqual(error.exception.errno, 22)
def test_set_uid(self) -> None:
"""set uid"""
get_nonce(self.d, None)
with self.assertRaises(OSError) as error:
set_uid(self.d, self.uid, self.signature)
self.assertEqual(error.exception.errno, 1)
@ -152,13 +154,13 @@ class TestInvalidSignature(DynamicBoostControlTest):
"""fetch a parameter"""
with self.assertRaises(OSError) as error:
process_param(self.d, PARAM_GET_SOC_PWR_CUR, self.signature)
self.assertEqual(error.exception.errno, 1)
self.assertEqual(error.exception.errno, 11)
def test_set_param(self) -> None:
"""set a parameter"""
with self.assertRaises(OSError) as error:
process_param(self.d, PARAM_SET_PWR_CAP, self.signature, 1000)
self.assertEqual(error.exception.errno, 1)
self.assertEqual(error.exception.errno, 11)
class TestUnFusedSystem(DynamicBoostControlTest):